HugeCTR Inference Parameter Server]
A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models

Yingcan Wei

Matthias Langer

Fan Yu

Minseok Lee

South Korea

Kingsley Liu

Jerry Shi

Joey Wang

Recommendation systems are of crucial importance for a variety of modern apps and web services, such as news feeds, social networks, e-commerce, search, etc. To achieve peak prediction accuracy, modern recommendation models combine deep learning with terabyte-scale embedding tables to obtain a fine-grained representation of the underlying data. Traditional inference serving architectures require deploying the whole model to standalone servers, which is infeasible at such massive scale.

In this paper, we provide insights into the intriguing and challenging inference domain of online recommendation systems. We propose the HugeCTR Hierarchical Parameter Server (HPS), an industry-leading distributed recommendation inference framework, that combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. Among other things, HPS features (1) a redundant hierarchical storage system, (2) a novel high-bandwidth cache to accelerate parallel embedding lookup on NVIDIA GPUs, (3) online training support and (4) light-weight APIs for easy integration into existing large-scale recommendation workflows. To demonstrate its capabilities, we conduct extensive studies using both synthetically engineered and public datasets. We show that our HPS can dramatically reduce end-to-end inference latency, achieving 5~62x speedup (depending on the batch size) over CPU baseline implementations for popular recommendation models. Through multi-GPU concurrent deployment, the HPS can also greatly increase the inference QPS.

<concept_desc>Computing methodologies Cooperation and coordination</concept_desc>
<concept_desc>Computing methodologies Distributed algorithms</concept_desc>
<concept_desc>Information systems Novelty in information retrieval</concept_desc>

<concept_desc>Computing methodologies Knowledge representation and reasoning</concept_desc>
<concept_desc>Information systems Retrieval models and ranking</concept_desc>

[500]Computing methodologies Cooperation and coordination
[500]Computing methodologies Distributed algorithms
[500]Information systems Novelty in information retrieval

[500]Computing methodologies Knowledge representation and reasoning
[500]Information systems Retrieval models and ranking

§ INTRODUCTION

Recommendation Systems (RS) are used in various apps and online services, such as news feeds, e-commerce, social networks, search, etc. To provide accurate predictions, state-of-the-art algorithms rely on embedding-based deep learning models. <ref> illustrates the typical architecture of a deep recommendation model (DLRM). The input consists of dense features (e.g., age, price, etc.) and sparse features (e.g., user ID, category ID, etc.). The sparse features are transformed into dense embedding vectors through lookup in an embedding table, so that the result from combining these with the dense features can be fed through some densely connected deep learning model (e.g., a MLP, transformer, etc. <cit.>) to predict the Click-Through Rate (CTR).

A typical deep recommendation model (DLRM).

Key distribution if recall statistics approximately follow a power law distribution.

Embeddings can consume a significant portion of the memory capacity in a data center. Often, a significant amount of time is spent to retrieve these embeddings from a centralized parameter server, which adds latency that delays downstream computations. Unlike in throughput-oriented training systems <cit.>, online inference systems are tightly constrained by latency requirements <cit.>. Thus, the embedding lookup speed is essential for deep recommendation model inference performance.

During inference, each mini-batch of data usually references tens of thousands of embeddings. Realizing the exhaustive search of each embedding by its key requires the parameter server to walk certain internal data structures. The lookup of individual embeddings from an embedding table is usually independent, and, thus, easily parallelizable. At the same time, modern GPU architectures allow scheduling thousands of threads to run concurrently, and their memory subsystems adopt special memory technology that provides higher bandwidth and throughput than equivalent CPU memories <cit.>. These features could make GPU architectures ideal for processing embedding vector lookup workloads.

Challenges. The size of embedding tables used in state-of-the-art recommendation models can be vast, often ranging from tens of giga- to several terabytes, which is well beyond the memory complement of most GPUs. Furthermore, batch sizes during online inference are usually too small to efficiently utilize the massively parallel-processing-optimized computational resources of just a single GPU. Hence, embedding lookup workloads require large amounts of GPU memory, but only few computational resources. This imbalance of requirements significantly deviates from the available hardware, and diminishes GPUs' attractiveness for use in inference systems. Therefore, most existing solutions decouple the embedding lookup operation from the dense computations (i.e., the remainder of the model), which are executed in the GPU, and move it to the CPU <cit.>. Thereby, they forfeit the memory bandwidth advantages of GPUs, while the CPU and the communication bandwidth between the CPU and the GPU becomes the primary bottleneck. As a result, the disproportionate processing capabilities of GPUs sit mostly idle in such setups (=resource waste).

Approach. It is usually not possible to retain all embedding tables entirely in GPU memory. However, empirical evidence for real-world recommendation datasets suggests that embedding key access during inference for CTR and other recommendation tasks often exhibits strong locality, and approximately follows the power law distribution <cit.>. Hence, a significant proportion of the embedding keys per mini-batch reference only a small set of hot embeddings. Caching such hot embeddings in the GPU memory, where the remainder of the model is processed, makes partial GPU-accelerated embedding lookup possible. Based on these observations, we have built an inference framework, namely the HugeCTR Hierarchical Parameter Server (HPS), to take advantage of GPU resources, without being constrained to GPU memory limitations. In particular, HPS introduces a GPU embedding cache data structure that tries to retain hot embeddings within the GPU memory. The cache is complimented by a parameter server that keeps a full copy of all embedding tables. Our contributions can be summarized as follows:

* Hierarchical database architecture that allows utilizing cluster memory resources, and provides an asynchronous update mechanism to maintain a high GPU embedding cache hit rate during online inference.

* High-performance dynamic GPU embedding cache that maximizes throughput by tracking and caching frequently occurring embeddings in high-throughput GPU memory, while overlapping the host/device transfers.

* Online model update mechanism for distributed inference deployments (i.e., real-time updates).

* Customizable HPS backend that provides concurrent model execution, hybrid model deployment, and ensemble model pipeline services for NVIDIA Triton GPU inference server <cit.>.

This paper is structured as follows. In <ref>, we provide a fundamental discussion of core concepts that underpin our approach. Then, we subsequently introduce and discuss the individual components of the HPS and how they interact in Sections <ref>~<ref>. In <ref>, we discuss how our HPS realizes real-time model updates. Eventually, we conduct an experimental study to evaluate the performance of the HPS in <ref>, and provide concluding remarks in <ref>.

§ BACKGROUND

§.§ Embedding Tables

Current mainstream algorithms in advertising, recommendation and search adopt model structures that combine embedding tables with a deep neural network to form a deep learning recommendation model (DLRM) <cit.>. At the foundation of such models are embeddings $e$, which represent learned numeric representations of user or item features as dense vectors that are aligned in some $d$-dimensional space ($e \in \mathbb{R}^d$). We let $E_j = \{ e_j^0, e_j^1, \cdots, e_j^n \}$ denote some discrete subset of the embeddings for some feature $j$. For easy access within the model, we organize these embeddings as embedding feature tables of the form

\begin{equation}
T_j = \langle K_j, E_j \rangle = \lbrace \langle k_j^0, e_j^0 \rangle, \langle k_j^1, e_j^1 \rangle, \cdots, \langle k_j^n, e_j^n \rangle \rbrace ~,
\end{equation}

that consist of tuples $\langle k_j^i, e_j^i \rangle$, where $k_j^i$ is a key that identifies and references the $i$-th embedding table entry $e_j^i$. The key space $K_j = \{ k_j^0, k_j^1, \cdots, k_j^n \}$ is discrete, and $(\forall k_j^i, k_j^z \in K_j)(i \neq z \rightarrow k_j^i \ne k_j^z)$ is implied. The value of each key depends on the underlying data or task. Usually, the key space is sparsely populated.

To evaluate a DLRM for CTR (cf. <ref>), the driver application must first select the entries from the embedding table that are relevant to make the prediction. This is simply done by looking up[Direct key lookup is the predominantly used method. For complex models other methods to determine the query keys $Q$ may exist.] the keys from the query key subset $Q_j$ for each embedding feature table (i.e., $Q = \{ Q_0 \subseteq K_0, Q_1 \subseteq K_1, \cdots \}$). Thus, $Q_j = \{ q_j^0, q_j^1, \cdots, q_j^m \}$ denotes a query for looking up the $m$ corresponding embeddings entries from $T_j$. The corresponding result set is $R_{Q_j} = \{ q_j^0 \mapsto e_j^0, q_j^1 \mapsto e_j^1, \cdots, q_j^m \mapsto e_j^m \}$. Accelerating the retrieval of such result sets at scale is our primary objective.

§.§ Deduplication and skewness.

To avoid unnecessary double-lookups if the same embedding table entries are required multiple times, HugeCTR always applies a deduplication operator prior to executing any subsequent steps (i.e., $Q^* = \text{\textsc{dedup}}(Q)$). This is particularly important for mini-batch processing, where $Q$ is the concatenation of many input samples. Naturally, deduplication becomes more effective if the skewness of the query distribution $\mathcal{Q}$ increases.

Understanding and utilizing skewness properties of the dataset is peril achieving peak efficiency. Many real-world recommendation datasets (e.g., Criteo <cit.>) exhibit a power law distribution <cit.>. That is, certain subsets of keys are referenced more frequently than others, such that sampling $q_j \sim \mathcal{Q}_j$ eventually approximates $p(x) \propto x^{-\alpha}$. <ref> depicts a scenario where the embedding key recall statistics approximate a power law distribution. The key space can be divided into three categories: (1) Frequent embeddings effectively appear in every batch. They represent a significant fraction of the recall/update requests. The frequent set is usually small. Even for large embedding corpora, only up to a few thousand embeddings appear that regularly. (2) Stochastic embeddings appear every few batches (i.e., somewhat regularly over time). (3) Rare embeddings are at the far end of the spectrum. They appear rather infrequently in queries.

Because requests repeatedly reference frequent and stochastic embeddings, applying efficient caching methods to them improves the overall system performance the most. Our HPS design (see <ref>) builds up on this observation.

Such category assignments of embeddings are absolutely determined if the query dataset is fixed. When training HugeCTR models, we take advantage of this to achieve world-class model convergence rates <cit.>. During online inference, the recall statistics depend on the actually incoming user requests. These cannot be preempted. Due to sudden events, changing trends or fashion, the category assignment of individual embeddings can vary over time. For most recommendation tasks, the runtime statistics are in a constant flux. Thus, inference systems must be adaptive.

§.§ GPU-accelerated Inference Architecture

Parameter servers for ML inference workloads mostly rely on database operations that are trivially parallelizable with GPUs <cit.>. Applications that require fast response times, e.g., online transaction processing (OLTP), often benefit greatly from GPU acceleration <cit.>. However, GPU memory constraints pose a tough challenge. To achieve scalability, many existing GPU-accelerated database systems, as well as our approach, implement a hierarchical storage architecture that extends the available GPU memory with other storage resources. Because external memory resources cannot be accessed as efficiently as native GPU memory <cit.>, the data exchange performance with the host system is emphasized in such systems <cit.>. To achieve peak performance, overlapped query processing must be used in conjunction with efficient communication patterns and data placement strategies that are actively refined at runtime <cit.>.

Constructing a parameter server for a machine learning platform poses many challenges <cit.>. When designing mixed GPU/CPU-based architectures for inference production environments, at least two major bottlenecks must be overcome:
(1) High latency due to DRAM bandwidth limitations when communicating between CPU and GPU <cit.>.
(2) Deployment latency due to growing model size and complexity induced by online training, because fast-paced incremental model updates pose a great challenge with respect to data consistency and bandwidth. To address these bottlenecks, our HPS is specifically tailored for deployment as an inference parameter server for large-scale recommendation models on GPUs. It handles the data synchronization and communication to share model parameters (embedding tables) across different inference nodes <cit.>, and performs various optimizations to improve GPU utilization during parallel multi-model/multi-GPU inference, including the organization of the distributed embedding table into partitions <cit.>, GPU-friendly caching <cit.>, and an asynchronous data movement mechanism <cit.>.

§ HIERARCHICAL PARAMETER SERVER

Our Hierarchical Parameter Server (HPS) allows HugeCTR to use models with huge embedding tables for inference. This is achieved through extending the embedding storage space beyond the constraints of GPUs using CPU memory resources from across the cluster. The design target of the HPS is to address the three challenges that traditional CPU parameter server approaches typically suffer from most:

* Downloading/streaming of model parameters from the centrally maintained embedding table partitions in CPU memory to the model instances on individual GPU compute devices. This issue is magnified if the embedding table cannot be fit entirely into the GPU memory. HPS greatly alleviates this problem through a GPU caching mechanism that takes advantage of the locality of the data distribution.

* Increased deployment cost caused by high-availability requirements of inference platforms and bandwidth limitations. By jointly organizing and using the distributed CPU memories of the inference cluster, the HPS saves resources and realizes immediate online model updating (i.e., training to inference updates).

* Parameter update and refresh between the GPU cache and the parameter server. This is particularly challenging if only a part of the model is loaded into GPU memory, so that parameters are missed on the GPU during lookup. HPS handles additional parameter exchanges between the CPU and GPU using an asynchronous insertion and refreshing mechanism to maintain parameter consistency.

§.§ Storage Architecture

Our HPS is implemented as a 3-level hierarchical memory architecture (cf. <ref>) that utilizes GPU GDDR and/or high-bandwidth memory (HBM), distributed CPU memory and local SSD storage resources. The communication mechanisms between these components ensure that the most frequently used embeddings reside in the GPU embedding cache. Somewhat frequently used embeddings are cached in CPU memory, while a full copy of all model parameters, including those that rarely occur, is always kept available on the hard disk/SSD. To minimize delays, we overlap parameter updating and the migration of missing parameters from higher storage levels (SSD $\rightarrow$ CPU memory $\rightarrow$ GPU memory) with the dense model computation. The three memory architecture levels of the HPS are defined as follows:

HugeCTR HPS storage architecture. Annotated process (-0.1em1~-0.1em5) depicts asynchronous model updating and embedding cache refresh in a distributed deployment (see <ref>).

a:ec-insertEmbedding cache insertion.

$Q$ = query keys;
$N$ = total size of embedding cache;
$t$ = hit rate threshold;
$E_Q$ = pre-allocated buffer to store embeddings.
$ws$ $\gets$ requestMemoryBlock()  Apply for workspace.
$Q^*$ $\gets$ dedup($Q$)  Deduplicate keys.
EmbCacheLookup($Q^*$)  GPU cache lookup keys.
$ws_\text{hitRate}$ $\gets$ $1 - \lvert ws_\text{missingKeys} \rvert \div N$
$ws_\text{hitRate} < t$
ParServerLookup($ws_\text{missingKeys}$)  Synchronous!
EmbCacheInsert($ws_\text{missingKeys}$, $ws_\text{vectors}$)
$E_Q$ $\gets$ $ws_\text{vectors}$
$E_Q$ $\gets$ $ws_\text{vectors}$   Fill in default values.
Queue asyncInsert() task.

GPU embedding cache (level 1). This is a dynamic cache designed for recommendation model inference. It attempts to improve the lookup performance for embeddings by reducing additional/repetitive parameter movements through cleverly utilizing data locality to keep frequently used features (i.e. the hot features) in the GPU memory. The GPU cache supports several operators (see <ref>), as well as a dynamic insertion and an asynchronous refresh mechanism (see <ref>) to retain a high cache hit rate.

Parameter partitions (level 2) store a partial copy of the embedding parameters in CPU memory. They act as an extension to the GPU embedding cache, and are queried if an embedding is required that is currently not present in the cache. Practitioners can choose between stand-alone deployment and cluster deployment, depending on their application scenario. In stand-alone deployment, the partitions are either placed in an optimized parallel hash-map (server-less deployment) or a local Redis instance. Distributed deployments can make use of multi-node Redis configurations. The contents of each partition are asynchronously adjusted in response to the queries processed by all inference nodes of a deployment. To receive online updates, parameter partitions can subscribe to topics from a distributed event stream.

Parameter replications (level 3). To ensure fault-tolerance, HPS retains a full copy of all model parameters (i.e., a model replica) in a disk-based RocksDB key-value store in each inference node. This fallback storage is accessed if a lookup request towards the corresponding parameter partitions fails. Thus, if given enough time budget, a HPS deployment is always able to produce a full answer to every query. To stay up to date, each node separately monitors the distributed event stream and applies online updates at its own pace.

§ INFERENCE GPU EMBEDDING CACHE

When processing online inference workloads, it is usually not possible to know which embedding table subsets will be required next. Therefore, our GPU embedding cache is designed as a general-purpose dynamic cache, which can accept new embeddings by evicting old embeddings.

§.§ Cache Data Model

The GPU embedding cache consists of hierarchical 3-level structures as shown in <ref>: slots, slabs and slabsets.

GPU embedding cache data model.

General data-flow in HugeCTR model deployments.

Slots represent the GPU embedding cache's basic storage unit. Each slot contains an embedding key, the associated embedding vector, and an access counter.

Slabs. Modern GPU architectures manage and execute code in warps (groups of 32 threads; <cit.>). Peak performance can be achieved by writing warp-aware programs. Therefore, we group 32 slots into one slab, so that each warp thread is assigned to a distinct slot. When searching for matching embedding keys, we use warps to linearly probe slabs. In determining if and where a key was found in a slab, we perform register-level intra-warp communications (shuffle, ballot, etc.) to eliminate branch and memory divergences.

Slabsets. Like cache lines are grouped into cache sets in N-way set-associative caches, slabs are packed into slabsets. To exploit the massively parallel computing power of GPUs, each embedding key is first mapped to a particular slabset, but may then occupy any slot in that slabset. This way, linear probing is confined to a single slabset, without conflicting with independent slabsets. A smaller slabset size can reduce key search latency, but also leads to increasing conflict misses.
It is important to find the optimal slabset size to balance these two factors. We empirically set the slabset size to 2 for contemporary NVIDIA GPU architectures such as Ampere. To maximize GPU resource utilization and inference concurrency, inference workers can share the same embedding cache. Race conditions are prevented by only granting a single warp exclusive access to a slabset for particular cache operations, such as query and replace. This approach also implicitly ensures thread safety. Because the total number of slabsets is usually much higher ($\ge$ millions) than the maximum number of warps per GPU ($\ge$ thousands), the mutual exclusion does not incur significant stalls.

§.§ GPU Embedding Cache API

The GPU embedding cache supports four APIs:

* Query (a:ec-query) retrieves embedding vectors for sets of embedding keys. Missing keys are returned as a list that can be used to attempt fetching these embeddings from the parameter partitions.

* Replace (a:ec-replace) tries to insert embeddings by filling up empty slots first. If the number of empty slots is insufficient, the least recently used (LRU) embeddings are replaced. Already existing embeddings are ignored.

* Update (a:ec-update) first determines the intersection set between the input keys and the already cached keys, and then replaces the corresponding embedding vectors.

* Dump batch outputs the embedding keys currently stored in the cache.

Query, Replace and Update share the same core algorithm (cf. Algorithms <ref>, <ref> and <ref>). For each key, the assigned processing warp will first locate the slabset that contains the key using a hash function. Then it linearly probes the slabs within this slabset to either find the matching key-slot, or determine an empty/replaceable slot for insertion (replace & update only). The Dump API is trivial in that it simply copies all keys currently in the cache to the CPU memory.

a:ec-queryEmbedding cache query.
$Q$ = query keys (to lookup);
$S$ = slabsets currently in GPU memory;
$g$ = global iteration count.
$R$ = retrieved embedding values;
$M$ = indices in $Q$ and query key value of missed embeddings.
[Query API only]global thread ID $=$ $0$
$g$ $\gets$ atomicAdd($g$, $1$)
$workQueue$ $\gets$ warpFetchKeys($Q$, $tasksPerWarp$)
$workQueue$ $\neq$ $\emptyset$
$q$ $\gets$ fetchNextKey($workQueue$)
$s$ $\gets$ slabsetHasher($q$) $\mod$ $\lvert S \rvert$  Find slabset.
$n_0$ $\gets$ slabHasher($q$) $\mod$ $\lvert s \rvert$  Select first slab.
lock($s$)  Ensure exclusive access to slabset.
[Iterate over slabset $s$.]$i$ $\gets$ $0$ to $\lvert s \rvert$

[Key missing.]searched $\forall slab$ $\in$ $s$
$M$ $\gets$ $M$ $\|$ $\langle \text{\textsc{indexOf}(}q\text{)}, q \rangle$  Record it!
break for
$slab$ $\gets$ warpReadOutSlab($(n_0 + i) \mod \lvert s \rvert$)
$q$ $\in$ $slab$.keys
Key hit; get value, refresh counter!
$R_{\text{\textsc{indexOf}(}q\text{)}}$ $\gets$ $slab$.valueWhere(key $=$ $q$)
$slab$.counterWhere(key $=$ $q$) $\gets$ $g$
break for
[Key missing.]Null $\in$ $slab$.keys
$M$ $\gets$ $M$ $\|$ $\langle \text{\textsc{indexOf}(}q\text{)}, q \rangle$  Record it!
break for

Remark: $\|$ is a thread-safe concatenation operator.

a:ec-replaceEmbedding cache replace.
$Q$ = keys (to be replaced);
$E$ = associated embeddings.

[Iterate over slabset $s$.]$i$ $\gets$ $0$ to $\lvert s \rvert$

[Key missing, slot full!]searched $\forall slab$ $\in$ $s$
$slot$ $\gets$ warpFindSlotLRU($s$, slot_counter)
warpWriteSlot($slot$, $q$, $E_{q}$, $g$) Replace!
break for
$slab$ $\gets$ warpReadOutSlab($(n_0 + i) \mod \lvert s \rvert$)
[Key hit; refresh counter!]$q$ $\in$ $slab$.keys
$slab$.counterWhere(key $=$ $q$) $\gets g$
break for
[Slot available; insert!]Null $\in$ $slab$.keys
warpWriteSlot($emptySlot$, $q$, $E_{q}$, $g$)
break for

a:ec-updateEmbedding cache update.
$Q$ = keys (to be updated);
$E$ = associated embeddings.

[Iterate over slabset $s$.]$i$ $\gets$ $0$ to $\lvert s \rvert$

[Key not found; ignore!]searched $\forall slab$ $\in$ $s$
break for
$slab$ $\gets$ warpReadOutSlab($(n_0 + i) \mod \lvert s \rvert$)
[Key hit; update value!]$q$ $\in$ $slab$.keys
$slab$.valueWhere(key $=$ $q$) $\gets$ $E_q$
break for
[Not found, ignore!]Null $\in$ $slab$.keys
break for

All APIs launch CUDA kernels that are executed asynchronously. I.e., the control flow is immediately returned to the CPU. Because they are thread safe on the slabset-level (see <ref>), concurrent invocation of all APIs is permissible. To avoid frequent CUDA kernel launches and improve GPU resource utilization, all APIs accept mini-batches as input. The respective input keys are fairly distributed to warps, and pushed into a warp work queue.

§.§ Embedding Insertion

For failed lookups (i.e., key is currently not present in the GPU embedding cache), a cache insertion operation is triggered to fetch the missed embeddings from the parameter partitions in the CPU memory or a replica on a local SSD. As shown in a:ec-insert, the HPS has two insertion modes, between which the GPU embedding cache switches based on the relation between the current cache hit rate and a user-defined hit rate threshold:

Asynchronous insertion is activated if the cache hit rate is higher than the predefined threshold. For any missing keys, the default embedding vectors whose values are user configurable are returned immediately. The actual embeddings are fetched asynchronously from higher-level storage into the GPU embedding cache to have them available for future queries. This lazy insertion mechanism ensures that the prediction accuracy loss is negligible with a high hit rate.

Synchronous insertion blocks the rest of the pipeline until the missed embeddings have been fetched. With a reasonable threshold, synchronous insertion usually occurs only during the warm-up stage, or after model updates.

§ CPU MEMORY AND SSD STORAGE LAYERS

To process models that scale beyond GPU memory capacity, in addition to the GPU embedding cache (<ref>), the HPS incorporates two additional layers in its storage hierarchy. These layers are constructed based on either system memory, SSDs or network storage, and are highly modularized to support various backend implementations.

Volatile database (VDB) layers (level 2 in <ref>) reside in volatile memory such as system memory, that requires traversal through a NVLink or the PCIe bus to access it from a GPU. In comparison to GPU memory, system memory can be extended at lower costs. To grow even further, VDBs can take advantage of multiple, low-latency system memories on a inference cluster. For example, using our RedisClusterBackend VDB template implementation, users can use distributed Redis instances as a storage backend for embeddings. Thus, VDB implementations can, but do not have to be limited to machine boundaries. To distribute the workload, VDBs organize embedding table storage in partitions. Partitions are non-overlapping subsets of an embedding table that are stored in the same physical location. They are sparsely populated in response to the inference queries processed by all nodes that share VDB access. The maximum size (=overflow margin) and amount of partitions per embedding table are configurable, and subject to a trade-off. More smaller partitions allow for smoother load balancing, but each partition adds a small processing overhead.

VDBs are operated as an asynchronous cache. If a GPU embedding cache reports missing keys, the HPS queries the VDB next. Analogous to the embedding cache, each VDB entry contains a timestamp indicating when the entry was last accessed. For embedding vectors that were successfully retrieved, the VDB asynchronously updates this timestamp after returning the result.
Missed embedding vectors are scheduled for insertion into the VDB to accelerate potential future queries. Thereby, the partition assignment of each embedding is fixed and determined by the XXH64-hash value <cit.> of the key. Insertions happen asynchronously to not stall pending lookup processes, and subsequently fill up the VDB partitions. Per-partition eviction policies determine what should be done if a partition exceeds its overflow margin. We implement multiple eviction policies. For example, the evict oldest policy finds and prunes infrequently accessed keys.

Persistent database (PDB) layers (level 3 in <ref>) use hard-disks or SSDs to permanently store entire embedding tables (i.e., all model parameters). As such, the PDB is helpful to improve the prediction accuracy for datasets that exhibit an extreme long-tail distribution. PDB layers can serve as backup and ultimate ground truth for any number of models. To avoid key collisions, PDB implementations form separate key namespaces for each unique embedding table.

Our template implementation maps embedding tables to column groups in a RocksDB database, stored on a local SSD in each inference node. Hence, the entire model data is replicated in each inference node. This way, we achieve maximum fault tolerance, because node failures will not impair the ability of other inference nodes to fully answer each query. Continued operation is possible, even if a failure in a neighbor node brings down an attached Redis VDB. Without the VDB as an intermediate cache, it can of course take somewhat longer until embedding vectors for missed keys are asynchronously migrated into GPU embedding cache (see also <ref>). However, assuming the GPU embedding cache can retain a high-enough hit rate, clients should only witness minor deviations in inference performance.

§ ONLINE MODEL UPDATING

Thus far, we have described how the HPS organizes resources to enable inference with pretrained models. In <ref>, we have highlighted this portion of the data-flow graph in red ($\rightarrow$). However, there exist many scenarios where recommendations depend on recent information (e.g., user interactions in social networks). After completing a training epoch incremental updates have to be propagated to all inference nodes for improved recommendations. Our HPS achieves this functionality using a dedicated online updating mechanism.

Volatile & persistent database update. Model training is resource intensive, and therefore conducted by a set of nodes that is distinct form the inference cluster. Training sets for HugeCTR models are split into files that maximize the locality in the embedding cache. The model is trained by sequentially loading these files into the cache and processing the training episodes. Our online updating mechanism wraps around HugeCTR model training. It is designed as an auxiliary process (blue [$\rightarrow$] data-flow graph in <ref>) that can be turned on and off at any point in time.

Once training progress has been made, the training nodes dump their updates to an Apache Kafka-based message buffer <cit.>. This is done via our Message Producer API, which handles serialization, batching, and the organization of updates into distinct message queues for each embedding table. Inference nodes that have loaded the affected model can use the corresponding Message Source API to discover and subscribe to these message queues. As indicated in <ref>, separate subscriptions can be created for different VDB partitions. This allows nodes that share a VDB to also share the update workload among them. If a node becomes unresponsive, its current assignment is shifted to other nodes.

Applying online updates inevitably adds overhead. Therefore, we allow updates to be consumed lazily by each node using a background process. The execution of the update process is aligned with other I/O requests. To control and adjust the impact on online inference, users can limit the update ingestion speed and frequency.

Through message buffer subscriptions, updates are guaranteed to be in order and complete. Hence, upon fully processing all pending messages (sync), the individual database levels are guaranteed to be consistent (i.e., we guarantee final consistency). The lazy nature in which we apply updates implies that slight inconsistencies during model update periods have to be expected. However, in practice this does not matter because learning rates for model retraining are usually very small. As long as the optimization process is reasonably smooth, the prediction performance should not diminish significantly <cit.>. Note that the same assumption also underpins the working principle of the GPU embedding cache's query API, which returns default embedding values for missed keys if the hit rate criteria is met (see <ref>). However, since no downtime is required to ingest updates, it is possible to achieve continuous model improvement, which makes HPS particularly suitable for usage with highly active data sources.

Asynchronous GPU embedding cache refresh. The GPU embedding cache needs to be readily available when an inference request arrives. Ongoing streaming of small updates from message buffers to the GPU embedding cache would create spontaneous GPU load-spikes that are hard to predict and could diminish response times. Thus, instead of ingesting updates directly from Kafka, we allow the GPU embedding cache to regularly poll the VDB/PDB for updates and replace embeddings if necessary. The refresh cycle is configurable to best fit the training schedule. When using online training, the GPU embedding cache can be configured to periodically (minutes, hours, etc.) refresh its contents. When using offline training, refreshes are triggered through signals sent by the Triton model management API <cit.>. <ref> illustrates the entire sequence until a model update becomes effective in the GPU embedding cache:

* Monitor message stream. Dispatch and apply updates to CPU memory partitions (VDB) and the SSD (PDB).

* Dump GPU embedding cache keys in batches (size is configurable) and write them into the dump key buffer.

* Lookup embedding keys, written to the dump key buffer, from the CPU memory partitions and/or the SDD,

* and copy the corresponding embedding key-vectors to the queried key-vector buffer.

* Download the queried key-vector buffer into the GPU device and refresh the GPU embedding cache.

§ PERFORMANCE EVALUATION

In this section, we showcase the performance of our HPS from several aspects, including the end-to-end inference throughput and latency. Further, we provide empirical analyses of GPU embedding cache and different database backends, and investigate the impact of online updates on the HPS performance.

§.§ Experiment Setup

Unless specified otherwise, all experiments are carried out on a cluster consisting of NVIDIA DGX A100 <cit.> nodes. Each node is equipped with two AMD EPYC 7742 CPUs, 2 TB of CPU memory, eight NVSwitch-interconnected NVIDIA A100 GPUs with 80 GB GPU memory each, and eight Mellanox CX6 InfiniBand adapters for inter-node communication.

To demonstrate the HPS’s capabilities, we use two publicly available and two synthetically generated datasets. For the public datasets, we trained a DLRM model to obtain an embedding table which is then used for inference.

Criteo 1 TB <cit.> is a large publicly available log of user click behavior in response to ads, containing 13 dense features and 26 sparse features. The final embedding table amounts to ~90 GB, with an embedding vector size of 128.

MovieLens <cit.> is a small publicly available dataset containing movie recommendations. It contains 3 sparse features, of which one is a multi-hot feature. The
embedding table is only ~20 MB large (embedding vector size = 128).

Synthetic dataset A mimics the properties of Criteo 1 TB dataset. However, the final embedding table amounts to 650 GB. In lieu of generating a huge training dataset to obtain the embedding, we generate the embedding first by randomly creating embedding vectors of size 128. Then, we use the resulting key range to generate an inference request dataset by randomly drawing keys from a power-law distribution with $\alpha$ = 1.2 (see <ref>).
In the resulting inference requests about 95% of the embedding table lookups reference 10% of the embedding table.

Synthetic dataset B is created in a similar way as Synthetic dataset A, but contains 9 dense features and 130 sparse features. Further, we decreased the number of unique keys in each sparse feature so that the embedding table size becomes 81 GB large (i.e., close to the size of embedding table for the Criteo 1 TB dataset).

§.§ Inference performance

§.§.§ Single-GPU single-instance deployment on Triton

In this section, we evaluate the performance of HPS running on the top of a NVIDIA Triton Inference Server <cit.>, in comparison with a PyTorch CPU implementation. To measure inference performance, we utilize Triton's performance analyzer <cit.>. For all datasets, the size of DLRM model's trained dense weights is at most 10 MB, which can be easily loaded into either CPU or GPU memory.

HPS configuration for Criteo 1 TB dataset.

Components     Parameter     Value

3*GPU Emb. Cache
GPU cache %             0.5

Hit rate threshold       0.8

# instances / GPU         1

3*Volatile Database
Type                     Hash map

Initial cache rate            1.0

# partitions                  16

Volatile and persistent database random insertion.

(VDB, 32 threads)
(VDB, 3 nodes)
(PDB, SSD disk)

10       245.3 MB/s       162.0 MB/s       52.1 MB/s

25       236.0 MB/s       138.9 MB/s       45.3 MB/s

50       237.4 MB/s       132.3 MB/s       40.2 MB/s

75       222.0 MB/s       120.9 MB/s       38.3 MB/s

100       203.9 MB/s       111.2 MB/s       37.4 MB/s

<ref> lists the configuration for the GPU embedding cache and the VDB. Frequent embeddings are kept in GPU memory. Since our test system has 2 TB CPU memory, we can increase the VDB capacity to fully cache the embedding table. In contrast, our PyTorch baseline keeps both the embedding table and the dense weights in the CPU memory.

Embedding cache refreshment.

1         5.152       0.064       194.20

5        25.177       0.184       198.62

10        50.262       0.335       198.96

20       100.530       0.641       198.95

40       200.345       1.19        199.73

<ref> compares the end-to-end inference performance of our HPS and PyTorch CPU <cit.> with the Criteo 1 TB dataset. We measure latency and throughput, while varying the batch size from 32 to 131,072. HPS significantly outperforms PyTorch CPU in terms of average latency per batch. Because GPU compute and memory resources can be exploited better, larger batch sizes lead to higher speedup. At the maximum batch size of 131,072, a 62x speedup is achieved. The throughput ranges from 2.4 million samples per second (batch size = 1,024) to 6.4 million samples per second (batch size = 131,072). In contrast, PyTorch CPU delivers at most 0.2 million samples per second (batch size = 2,048). It is also worth noting that HPS also has a 2.35x throughput advantage when compared to a TensorFlow GPU inference solution <cit.> at batch size = 2,048 (1.43 million samples per second; model size = 15.6 GB).

For the Synthetic dataset A we keep all settings the same, except the GPU cache percentage, which we lowered to 5%, so that up to 32.5 GB of embeddings reside in GPU memory. Results are shown in <ref>. Like with Criteo, the throughput increases with the batch size and saturates around 131,072, while the latency remains stable.

width = , height = ,
xlabel = Batch Size,
xtick = data, symbolic x coords =
32, 64, 256, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072
xtick style = draw=none,
x tick label style=rotate=45,xshift=-0.5em,yshift=0.3em,
enlarge x limits = 0.075,
ylabel style = align=center,
ylabel = Avg. Latency (  ),
ymin = 0, ymax = 1400,
ytick = 200, 400, 600, 800, 1000, 1200,
yticklabels =
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
legend style =
at=(0.5,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em
legend cell align = left
color=LineColorA!20!ShadeColorA,ultra thick,
mark=*,mark options=LineColorA, scale=0.5
] coordinates
(32,      0.258)[ 0.258]
(64,      0.248)[ 0.248]
(256,     0.267)[ 0.267]
(1024,    0.417)[ 0.417]
(2048,    0.609)[ 0.609]
(4096,    0.973)[ 0.973]
(8192,    1.661)[ 1.661]
(16384,   2.867)[ 2.867]
(32768,   5.460)[ 5.460]
(65536,  10.248)[10.248]
(131072, 20.072)[20.072]
color=LineColorB!20!ShadeColorB,ultra thick,
mark=*,mark options=LineColorB,scale=0.5
] coordinates
(32,        1.313)[   1.313]
(64,        1.508)[   1.508]
(256,       4.975)[   4.975]
(1024,      7.130)[   7.130]
(2048,     11.185)[  11.185]
(4096,     24.852)[  24.852]
(8192,     55.278)[  55.278]
(16384,   128.639)[ 128.639]
(32768,   278.854)[ 278.854]
(65536,   571.534)[ 571.534]
(131072, 1246.993)[1246.993]

width = , height = ,
ybar = -0.7em,
bar width = 1.1em,
xlabel = Batch Size,
xtick = data, symbolic x coords =
32, 64, 256, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072
xtick style = draw=none,
x tick label style=rotate=45,xshift=-0.5em,yshift=0.6em,
enlarge x limits = 0.075,
ylabel style = align=center,
ylabel = Throughput (  ),
ymin = 0,
ytick = 1000, 2000, 3000, 4000, 5000, 6000,
yticklabels =
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
legend style =
at=(0.5,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);
legend cell align = left
[draw=LineColorA,fill=ShadeColorA,text=LineColorA] coordinates
(32,      124.031)[ 124.031]
(64,      258.064)[ 258.064]
(256,     958.801)[ 958.801]
(1024,   2455.635)[2455.635]
(2048,   3362.889)[3362.889]
(4096,   4209.660)[4209.660]
(8192,   4931.968)[4931.968]
(16384,  5714.684)[5714.684]
(32768,  6001.465)[6001.465]
(65536,  6395.003)[6395.003]
(131072, 6530.091)[6530.091]
[draw=LineColorB,fill=ShadeColorB,text=LineColorB] coordinates
(32,      24.354)[ 24.354]
(64,      42.423)[ 42.423]
(256,     51.448)[ 51.448]
(1024,   143.607)[143.607]
(2048,   183.097)[183.097]
(4096,   164.811)[164.811]
(8192,   148.194)[148.194]
(16384,  127.364)[127.364]
(32768,  117.509)[117.509]
(65536,  114.666)[114.666]
(131072, 105.110)[105.110]

Criteo 1 TB dataset (left: latency, right: throughput).

width = , height = ,
ybar = 0.45em,
bar width = 1.1em,
xlabel = Batch Size,
xtick = data, symbolic x coords =
32, 64, 256, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072
xtick style = draw=none,
x tick label style=rotate=45,xshift=-0.5em,yshift=0.6em,
enlarge x limits = 0.075,
axis y line* = left,
ylabel = Throughput (  ),
ymin = 0, ymax = 7000,
ytick = 1000, 2000, 3000, 4000, 5000, 6000,
yticklabels =
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
legend style =
at=(0.33,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);

[draw=LineColorA,fill=ShadeColorA,text=LineColorA] coordinates
(32,      128.000)[ 128.000]
(64,      259.109)[ 259.109]
(256,     888.888)[ 888.888]
(1024,   2085.539)[2085.539]
(2048,   2860.335)[2860.335]
(4096,   3908.396)[3908.396]
(8192,   4584.219)[4584.219]
(16384,  5183.169)[5183.169]
(32768,  5521.145)[5521.145]
(65536,  5719.172)[5719.172]
(131072, 5816.890)[5816.890]

width = , height = ,
axis x line = none,
xlabel = Batch size,
xtick = data, symbolic x coords =
32, 64, 256, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072
xtick style = draw=none,
enlarge x limits = 0.075,
axis y line* = right,
ylabel style = align=center,
ylabel = Avg. Latency (  ),
ylabel style = align=center, rotate=-180,
ymin = 0, ymax = 70,
ytick = 10, 20, 30, 40, 50, 60,
yticklabels =
major tick length=0.3em,
legend style =
at=(0.67,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em

color=LineColorC!50!White,very thick,
mark=*,mark options=LineColorC!90!White, scale=0.5
] coordinates
(32,      0.250)[ 0.250]
(64,      0.247)[ 0.247]
(256,     0.288)[ 0.288]
(1024,    0.491)[ 0.491]
(2048,    0.716)[ 0.716]
(4096,    1.048)[ 1.048]
(8192,    1.787)[ 1.787]
(16384,   3.161)[ 3.161]
(32768,   5.935)[ 5.935]
(65536,  11.459)[11.459]
(131072, 22.533)[22.533]

Synthetic dataset A.

Single-instance performance.

§.§.§ Multi-GPU multi-instance deployment on Triton

To demonstrate how HPS can take advantage of multi-GPU environments, we set the batch size to 1,024 and test with both, Criteo 1 TB and the synthetic dataset A. We program the Triton Inference Server to evenly distribute inference instances while varying the number of GPUs <cit.>. <ref> shows the resulting average QPS. With a single GPU, the QPS improves until up to 4 instances are deployed. This is due to enhanced GPU resource utilization from sharing the GPU embedding cache concurrently. Beyond 4 instances, increased resource contention degrades the QPS. Contention can be amortized by deploying the same number of instances on more GPUs. Consequently, the highest QPS (7.2x speedup) is achieved when deploying 8 model instances on 8 GPUs, so that each GPU has its own embedding cache. Note that they may still share VDB parameter partitions. To summarize, when deploying multiple instances,
using both scaling up per GPU and multi-GPU scale out maximizes the QPS.

§.§.§ Warm-up and stable stage performance of the GPU embedding cache

To achieve stable performance, the HPS has to pass the warm-up stage, during which hot embeddings are fetched into the GPU embedding cache. During the warm-up stage, the hit rate keeps increasing until the cache is fully occupied. Thereby, the hit rate threshold controls whether cache updates should be applied either in synchronous or asynchronous mode (see <ref>).

width = , height = ,
xlabel = Iteration #,
enlarge x limits = 0.025,
axis y line* = left,
ylabel style = align=center,
ylabel = Latency (  ),
ymode = log,
ymin = 0.4, ymax=20,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
legend style =
at=(0.5,1.015), anchor=south,
inner sep=0,
legend columns=2,
/tikz/every even column/.append style=column sep=1.4em
legend cell align = left
Latency ();
color=LineColorA,very thick
] table [x=x, y=lat-0, col sep=comma] results-7-2-3-crit.csv;
Hit rate ();
[color=LineColorD,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;
Latency ();
color=LineColorB,very thick
] table [x=x, y=lat-0-5, col sep=comma] results-7-2-3-crit.csv;
Hit rate ();
[color=LineColorE,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;
Latency ();
color=LineColorC,very thick
] table [x=x, y=lat-1, col sep=comma] results-7-2-3-crit.csv;
Hit rate ();
[color=LineColorF,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;

width = , height = ,
axis x line = none,
enlarge x limits = 0.025,
axis y line* = right,
ylabel style = align=center, rotate=-180,
ylabel = Hit Rate (  ),
ymin = 0, ymax = 0.9,
ytick = 0.2, 0.4, 0.6, 0.8,
yticklabels = 20, 40, 60, 80,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,loosely dashed,Gray
color=LineColorD,very thick
] table [x=x, y=hit-0, col sep=comma] results-7-2-3-crit.csv;
color=LineColorE,very thick
] table [x=x, y=hit-0-5, col sep=comma] results-7-2-3-crit.csv;
color=LineColorF,very thick
] table [x=x, y=hit-1, col sep=comma] results-7-2-3-crit.csv;

Warm-up inference latency for the Criteo 1 TB dataset.

width = , height = ,
xlabel = Iteration #,
enlarge x limits = 0.025,
axis y line* = left,
ylabel style = align=center,
ylabel = Latency (  ),
ymode = log,
ymin = 0.4, ymax=20,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
legend style =
at=(0.5,1.015), anchor=south,
inner sep=0,
legend columns=2,
/tikz/every even column/.append style=column sep=1.4em
legend cell align = left
Latency ();
color=LineColorA,very thick
] table [x=x, y=lat-0, col sep=comma] results-7-2-3-syn.csv;
Hit rate ();
[color=LineColorD,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;
Latency ();
color=LineColorB,very thick
] table [x=x, y=lat-0-5, col sep=comma] results-7-2-3-syn.csv;
Hit rate ();
[color=LineColorE,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;
Latency ();
color=LineColorC,very thick
] table [x=x, y=lat-1, col sep=comma] results-7-2-3-syn.csv;
Hit rate ();
[color=LineColorF,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;

width = , height = ,
axis x line = none,
enlarge x limits = 0.025,
axis y line* = right,
ylabel style = align=center, rotate=-180,
ylabel = Hit Rate (  ),
ymin = 0, ymax = 0.7,
ytick = 0.2, 0.4, 0.6,
yticklabels = 20, 40, 60,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,loosely dashed,Gray
color=LineColorD,very thick
] table [x=x, y=hit-0, col sep=comma] results-7-2-3-syn.csv;
color=LineColorE,very thick
] table [x=x, y=hit-0-5, col sep=comma] results-7-2-3-syn.csv;
color=LineColorF,very thick
] table [x=x, y=hit-1, col sep=comma] results-7-2-3-syn.csv;

Warm-up inference latency for the Synthetic dataset A.

width = , height = ,
axis y line* = right,
xlabel = Interation #,
xmin = 1025, xmax = 9993,
xtick = 2048, 4096, 6144, 8192,
x tick label style =
/pgf/number format/.cd,
scaled x ticks=false,
1000 sep=
enlarge x limits = 0.025,
ylabel = Hit Rate (  ),
ylabel style = align=center, rotate=180,
ymin = 0.3, ymax = 0.95,
ytick = 0.4, 0.6, 0.8, yticklabels = 40, 60, 80,
major tick length=0.3em,
ymajorgrids = true,
major grid style=thin,dotted,Gray,
legend style =
inner sep=0,
legend columns=2,
/tikz/every even column/.append style=column sep=0.9em
legend cell align = left
Latency ();
[color=LineColorA,thick] coordinates  (0, 0) (1, 1) ;
Hit rate ();
] table [x=idx, y=ds-hr-0.01, col sep=comma] cache-rate-7-2.csv;
Latency ();
[color=LineColorB,thick] coordinates  (0, 0) (1, 1) ;
Hit rate ();
] table [x=idx, y=ds-hr-0.05, col sep=comma] cache-rate-7-2.csv;
Latency ();
[color=LineColorC,thick] coordinates  (0, 0) (1, 1) ;
Hit rate ();
] table [x=idx, y=hr-syn, col sep=comma] cache-rate-7-2.csv;

width = , height = ,
axis x line = none,
axis y line* = left,
xmin = 1025, xmax = 9993,
enlarge x limits = 0.025,
ylabel = Latency (  ),
ylabel style = align=center,
ymin = 0.5, ymax = 3.75,
ytick = 1, 2, 3,
major tick length=0.3em,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,loosely dashed,Gray
] table [x=idx, y=ds-lat-0.01, col sep=comma] cache-rate-7-2.csv;
] table [x=idx, y=ds-lat-0.05, col sep=comma] cache-rate-7-2.csv;
] table [x=idx, y=lat-syn, col sep=comma] cache-rate-7-2.csv;

Stable stage inference latency for the Synthetic dataset A.

Inference latency during warm-up and stable stage.

First, we study the GPU embedding cache's behavior during the warm-up stage. Figures <ref> (Criteo 1 TB) and <ref> (Synthetic dataset A) respectively show how the hit rate and inference latency change as the inference session progresses (batch size = 1,024). Using a hit rate threshold of 0.0, the inference latency stabilizes very quickly because the cache is always updated asynchronously (i.e., lazily). When setting the threshold to 1.0, the stabilization period is much longer. The overall latency is higher because cache updates block the inference pipeline. With a hit rate threshold of 0.5, the latency is at first relatively high because of blocking updates. Once the hit rate threshold is met, the latency is lowered and flattens overall. In other words, properly setting the hit rate threshold allows taking advantage of both blocking and asynchronous updates to balance the latency and hit rate.

Next, we analyze the further development of the latency as the cache enters the stable stage, where its hit rate saturates, using the synthetic inference request data. For this experiment, the batch size and hit rate threshold are both fixed at respectively 1,024 and 1.0. Results are shown in <ref>. We measured with GPU cache percentage ratios of 1% and 5%. Note that cutting the cache percentage by a fifth degrades the saturated hit rate from 76% to 70%, but leads only to a mean latency increase of 5%. Thus, due to the HPS's effective usage of the request data's skewness property, a high performance can be retained, without having the embedding cache occupy too much GPU memory. The distribution of the input request data  also affects the cache performance. Using input request data with highly amplified locality that we generated just for this experiment ( in <ref>) and a GPU cache percentage ratio of 5%, the hit rate will eventually saturate at 100%. The latency improves accordingly. As the hit rate surpasses 90%, the latency is already 23% lower than with the unaltered Synthetic dataset A.

§.§.§ End-to-end inference performance

To verify its overall effectiveness, we study the end-to-end inference throughput performance of the HPS when combining differently configured storage layers (see <ref>). Results for the Criteo 1 TB dataset are presented in <ref>. We can draw the following conclusions: (1) HPS provides better inference performance with large batch sizes. (2) Unsurprisingly, the best performfance can be obtained by caching the entire embedding table. However, HPS still attains comparable performance figures for large models that cannot fit into memory. (3) Reducing the GPU cache ratio (20% $\rightarrow$ 10%), while increasing the maximum size of the Redis VDB (40% $\rightarrow$ 45%) yields better inference performance. Thus, the VDB as a 2nd-level cache can lower the pressure on the GPU cache, while the HPS-aware GPU cache update mechanism ensures a high hit rate, and greatly reduces the inference latency.

The advantages of having a sophisticated hierarchical storage architecture become more remarkable as the model size increases. <ref> shows that granting the HPS just 5% more CPU memory resources leads to an 1.24x end-to-end throughput increase with the much larger Synthetic dataset A.

width = 10cm, height = 5cm,
ybar = 0.3em,
bar width = 0.8em,
xlabel = Criteo 1 TBSynthetic dataset A,
xtick = data, symbolic x coords =
crt-1, crt-2, crt-4, crt-8, syn-1, syn-2, syn-4, syn-8
xticklabels =
1 GPU, 2 GPUs, 4 GPUs, 8 GPUs,
1 GPU, 2 GPUs, 4 GPUs, 8 GPUs
major tick length = 0,
minor x tick num = 1,
minor tick length = 1em,
xtick pos = left,
enlarge x limits = 0.1,
ylabel style = align=center,
ylabel = QPS (),
ymin = 0, ymax = 6250,
ytick = 1000, 3000, 5000,
yticklabels = 1 k, 3 k, 5 k,
ymajorgrids = true,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
nodes near coords =

[1000 sep=,precision=0]

nodes near coords align = vertical,
nodes near coords style=font=,
legend style =
at=(0.5,1.015), anchor=south,
inner sep=0,
legend columns=4,
/tikz/every even column/.append style=column sep=2em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);
legend cell align=left
1 Instance;
[draw=LineColorA, fill=ShadeColorA, text=LineColorA] coordinates
(crt-1, 1700.4)[1700.4]
(crt-2, -0.1)[0.0]
(crt-4, -0.1)[0.0]
(crt-8, -0.1)[0.0]
(syn-1, 1762.0)[1762.0]
(syn-2, -0.1)[0.0]
(syn-4, -0.1)[0.0]
(syn-8, -0.1)[0.0]
2 Instances;
[draw=LineColorB, fill=ShadeColorB, text=LineColorB] coordinates
(crt-1, 2516.4)[2516.4]
(crt-2, 2941.2)[2941.2]
(crt-4, -0.1)[0.0]
(crt-8, -0.1)[0.0]
(syn-1, 2280.2)[2280.2]
(syn-2, 2378.0)[2378.0]
(syn-4, -0.1)[0.0]
(syn-8, -0.1)[0.0]
4 Instances;
[draw=LineColorC, fill=ShadeColorC, text=LineColorC] coordinates
(crt-1, 3235.4)[3235.4]
(crt-2, 4046.4)[4046.4]
(crt-4, 4317.2)[4317.2]
(crt-8, -0.1)[0.0]
(syn-1, 3129.6)[3129.6]
(syn-2, 4004.0)[4004.0]
(syn-4, 4345.6)[4345.6]
(syn-8, -0.1)[0.0]
8 Instances
[draw=LineColorD, fill=ShadeColorD, text=LineColorD] coordinates
(crt-1,  787.4)[ 787.4]
(crt-2, 1093.4)[1093.4]
(crt-4, 4747.8)[4747.8]
(crt-8, 5669.8)[5177.8]
(syn-1, 2819.2)[2819.2]
(syn-2, 2839.4)[2839.4]
(syn-4, 5177.8)[5177.8]
(syn-8, 5489.2)[5489.2]

(rel axis cs: 0.5835, 0.0) – (rel axis cs: 0.5835, -0.125);

Multi-GPU, multi-instance batch throughput.

width = , height = ,
xlabel = Iteration #,
enlarge x limits = 0.025,
axis y line* = left,
ylabel style = align=center,
ylabel = Accuracy (  ),
ymin = 0.2, ymax = 1,
ytick = 0.4, 0.6, 0.8,
yticklabels = 40, 60, 80,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
legend style =
at=(0.5,1.015), anchor=south,
inner sep=0,
legend columns=2,
/tikz/every even column/.append style=column sep=1.4em
legend cell align = left
Accuracy ();
] table [x=x, y=acc-0, col sep=comma] acc-hit-rate-crit.csv;
Hit rate ();
[color=LineColorD,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;
Accuracy ();
] table [x=x, y=acc-0-5, col sep=comma] acc-hit-rate-crit.csv;
Hit rate ();
[color=LineColorE,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;
Accuracy ();
] table [x=x, y=acc-1, col sep=comma] acc-hit-rate-crit.csv;
Hit rate ();
[color=LineColorF,very thick] coordinates  (0, 0) (0.0001, 0.0001) ;

width = , height = ,
axis x line = none,
enlarge x limits = 0.025,
axis y line* = right,
ylabel style = align=center, rotate=-180,
ylabel = Hit Rate (  ),
ymin = 0.2, ymax = 1,
ytick = 0.4, 0.6, 0.8,
yticklabels = 40, 60, 80,
ymajorgrids = false,
major tick length=0.3em,
major grid style=thin,loosely dashed,Gray
] table [x=x, y=hit-0, col sep=comma] acc-hit-rate-crit.csv;
] table [x=x, y=hit-0-5, col sep=comma] acc-hit-rate-crit.csv;
] table [x=x, y=hit-1, col sep=comma] acc-hit-rate-crit.csv;

Criteo 1 TB: Accuracy and hit rate during inference.

width = 13cm, height = ,

ybar = 0.45em,
bar width = 1.85em,
xlabel = Batch Size,
xtick = data, symbolic x coords = 1024, 2048, 4096, 8192,
xtick style = draw=none,
enlarge x limits = 0.18,
ylabel style = align=center,
ylabel = Throughput (  ),
ymode = log, log basis y = 10,

ymax = 4500,
ytick = 250, 500, 1000, 2000,
yticklabels = 250k, 500k, 1M, 2M,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
nodes near coords =

/ 1000

nodes near coords align = vertical,
nodes near coords style=font=,
legend style =
at=(1,1.015), anchor=south east,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=1em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);

10% EC + 45% Redis + RocksDB;
[draw=LineColorA, fill=ShadeColorA, text=LineColorA] coordinates
(1024,  775.133711)[ 775.133711]
(2048,  869.640838)[ 869.640838]
(4096, 1126.760563)[1126.760563]
(8192, 1273.398701)[1273.398701]
20% EC + 40% Redis + RocksDB;
[draw=LineColorB, fill=ShadeColorB, text=LineColorB] coordinates
(1024, 536.020583)[ 536.020583]
(2048, 746.491490)[ 746.491490]
(4096, 753.012048)[ 753.012048]
(8192, 876.501008)[ 876.501008]
20% EC + PHM;
[draw=LineColorC, fill=ShadeColorC, text=LineColorC] coordinates
(1024, 1166.316772)[1166.316772]
(2048, 1970.831691)[1970.831691]
(4096, 2766.251729)[2766.251729]
(8192, 3137.293623)[3137.293623]
20% EC + Redis;
[draw=LineColorD, fill=ShadeColorD, text=LineColorD] coordinates
(1024,  838.293235)[ 838.293235]
(2048, 1194.600406)[1194.600406]
(4096, 1297.521733)[1297.521733]
(8192, 1701.170422)[1701.170422]
20% EC + RocksDB;
[draw=LineColorE, fill=ShadeColorE, text=LineColorE] coordinates
(1024, 222.464461)[ 222.464461]
(2048, 246.445030)[ 246.445030]
(4096, 201.641361)[ 201.641361]
(8192, 259.652585)[ 259.652585]

Criteo 1 TB dataset; PHM = Parallel HashMap.

width = , height = ,

ybar = 0.45em,
bar width = 1.85em,
xlabel = Batch Size,
xtick = data, symbolic x coords = 1024, 2048, 4096, 8192,
xtick style = draw=none,
enlarge x limits = 0.18,
ylabel style = align=center,
ylabel = Throughput (  ),
ymin = 350, ymax = 650,
ytick = 400, 500, 600,
yticklabels = 0.4M, 0.5M, 0.6M,
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
nodes near coords = [precision=0]k,
nodes near coords align = vertical,
nodes near coords style=font=,
legend style =
at=(1,1.015), anchor=south east,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=1em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);

5% EC + 20% Redis + RocksDB;
[draw=LineColorA, fill=ShadeColorA, text=LineColorA] coordinates
(1024, 408.945687)[408.945687]
(2048, 458.987)[458.987]
(4096, 468.1142857)[468.114285]
(8192, 477.2780238)[477.278023]
5% EC + 25% Redis + RocksDB;
[draw=LineColorB, fill=ShadeColorB, text=LineColorB] coordinates
(1024, 502.206964)[502.206964]
(2048, 573.733751)[573.733751]
(4096, 584.308131)[584.308131]
(8192, 591.480144)[591.480144]

Synthetic dataset A.

HPS end-to-end inference throughput. Redis refers to a VDB with 40 storage partitions spread across a 3-node Redis cluster.

width = , height = ,
ybar = 0.45em,
bar width = 1.1em,
xlabel = Batch Size,
xtick = data, symbolic x coords =
32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
xtick style = draw=none,
x tick label style=rotate=45,xshift=-0.5em,yshift=0.6em,
enlarge x limits = 0.075,
axis y line* = right,
ylabel style = align=center, rotate=-180,
ylabel = Hit rate (  ),
ymin = 0.9825, ymax = 0.99625,
ytick = 0.985,0.9875,0.990,0.9925,0.995,
yticklabels =
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
legend style =
at=(0.11,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);

Hit rate;
[draw=LineColorD,fill=ShadeColorD,text=LineColorD] coordinates
(32,    0.985042)[0.985042]
(64,    0.984914)[0.984914]
(128,   0.986809)[0.986809]
(256,   0.985998)[0.985998]
(512,   0.984488)[0.984488]
(1024,  0.984717)[0.984717]
(2048,  0.985610)[0.985610]
(4096,  0.985921)[0.985921]
(8192,  0.989462)[0.989462]
(16384, 0.990117)[0.990117]
(32768, 0.991396)[0.991396]
(65536, 0.990771)[0.990771]

width = , height = ,
axis x line = none,
xlabel = Batch size,
xtick = data, symbolic x coords =
32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
xtick style = draw=none,
enlarge x limits = 0.075,
axis y line* = left,
ylabel style = align=center,
ylabel = Avg. Latency (  ),
ymin = 0, ymax = 5.5,
ytick = 1, 2, 3, 4, 5,
major tick length=0.3em,
legend style =
at=(0.62,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em

color=LineColorA!50!White,very thick,
mark=*,mark options=LineColorA!90!White, scale=0.5
] coordinates
(32,    0.19164)[0.19164]
(64,    0.241  )[0.241  ]
(128,   0.423  )[0.423  ]
(256,   0.437  )[0.437  ]
(512,   0.473  )[0.473  ]
(1024,  0.472  )[0.472  ]
(2048,  0.458  )[0.458  ]
(4096,  0.5825 )[0.5825 ]
(8192,  0.8463 )[0.8463 ]
(16384, 1.541  )[1.541  ]
(32768, 2.811  )[2.811  ]
(65536, 5.058  )[5.058  ]
color=LineColorF!70!White,very thick,
mark=*,mark options=LineColorF, scale=0.5
] coordinates
(32,    0.168   )[0.168   ]
(64,    0.180826)[0.180826]
(128,   0.19384 )[0.19384 ]
(256,   0.211956)[0.211956]
(512,   0.231457)[0.231457]
(1024,  0.258   )[0.258   ]
(2048,  0.315314)[0.315314]
(4096,  0.449116)[0.449116]
(8192,  0.681746)[0.681746]
(16384, 1.241128)[1.241128]
(32768, 2.195967)[2.195967]
(65536, 3.903513)[3.903513]
color=LineColorC!50!White,very thick,
mark=*,mark options=LineColorC!90!White, scale=0.5
] coordinates
(32,    0.167002)[0.167002]
(64,    0.183343)[0.183343]
(128,   0.20196 )[0.20196 ]
(256,   0.216294)[0.216294]
(512,   0.234857)[0.234857]
(1024,  0.281811)[0.281811]
(2048,  0.450771)[0.450771]
(4096,  0.541947)[0.541947]
(8192,  0.747654)[0.747654]
(16384, 1.279079)[1.279079]
(32768, 2.347651)[2.347651]
(65536, 3.633   )[3.633   ]

MovieLens dataset

width = , height = ,
ybar = 0.45em,
bar width = 1.1em,
xlabel = Batch Size,
xtick = data, symbolic x coords =
32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
xtick style = draw=none,
x tick label style=rotate=45,xshift=-0.5em,yshift=0.6em,
enlarge x limits = 0.075,
axis y line* = right,
ylabel style = align=center, rotate=-180,
ylabel = Hit rate (  ),
ymin = 0.905, ymax = 0.995,
ytick = 0.92,0.935,0.95,0.965,0.98,
yticklabels =
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
legend style =
at=(0.11,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);

Hit rate;
[draw=LineColorD,fill=ShadeColorD,text=LineColorD] coordinates
(32,    0.9809667892)[0.9809667892]
(64,    0.9799750228)[0.9799750228]
(128,   0.9772611092)[0.9772611092]
(256,   0.9740196099)[0.9740196099]
(512,   0.9699425205)[0.9699425205]
(1024,  0.9643837864)[0.9643837864]
(2048,  0.9569936306)[0.9569936306]
(4096,  0.9470993394)[0.9470993394]
(8192,  0.9349061848)[0.9349061848]

width = , height = ,
axis x line = none,
xlabel = Batch size,
xtick = data, symbolic x coords =
32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
xtick style = draw=none,
enlarge x limits = 0.075,
axis y line* = left,
ylabel style = align=center,
ylabel = Avg. Latency (  ),
ymin = 0, ymax = 6,
ytick = 1, 2, 3, 4, 5,
major tick length=0.3em,
legend style =
at=(0.62,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em

color=LineColorA!50!White,very thick,
mark=*,mark options=LineColorA!90!White, scale=0.5
] coordinates
(32,     0.4698455598)[ 0.4698455598]
(64,     0.4807016679)[ 0.4807016679]
(128,    0.4485380952)[ 0.4485380952]
(256,    0.5950644491)[ 0.5950644491]
(512,    0.6486721596)[ 0.6486721596]
(1024,   0.9242320048)[ 0.9242320048]
(2048,   1.640779661 )[ 1.640779661 ]
(4096,   2.826187654 )[ 2.826187654 ]
(8192,   5.073014553 )[ 5.073014553 ]
color=LineColorF!70!White,very thick,
mark=*,mark options=LineColorF, scale=0.5
] coordinates
(32,     0.3444728916)[ 0.3444728916]
(64,     0.6910228311)[ 0.6910228311]
(128,    0.6729395712)[ 0.6729395712]
(256,    0.6082594059)[ 0.6082594059]
(512,    0.6321461412)[ 0.6321461412]
(1024,   0.7191909091)[ 0.7191909091]
(2048,   1.264726445 )[ 1.264726445 ]
(4096,   1.605787879 )[ 1.605787879 ]
(8192,   2.72942804  )[ 2.72942804  ]
color=LineColorC!50!White,very thick,
mark=*,mark options=LineColorC!90!White, scale=0.5
] coordinates
(32,     0.2875819113)[ 0.2875819113]
(64,     0.3351384892)[ 0.3351384892]
(128,    0.4691317158)[ 0.4691317158]
(256,    0.5187517986)[ 0.5187517986]
(512,    0.5359193878)[ 0.5359193878]
(1024,   0.5811115108)[ 0.5811115108]
(2048,   0.8193977747)[ 0.8193977747]
(4096,   1.161410072 )[ 1.161410072 ]
(8192,   1.885827664 )[ 1.885827664 ]

Criteo 1 TB dataset

width = , height = ,
ybar = 0.45em,
bar width = 1.1em,
xlabel = Batch Size,
xtick = data, symbolic x coords =
32, 64, 128, 256, 512, 1024, 2048, 4096, 8192
xtick style = draw=none,
x tick label style=rotate=45,xshift=-0.5em,yshift=0.6em,
enlarge x limits = 0.075,
axis y line* = right,
ylabel style = align=center, rotate=-180,
ylabel = Hit rate (  ),
ymin = 0.84, ymax = 0.96,
ytick = 0.87,0.9,0.93,
yticklabels =
ymajorgrids = true,
major tick length=0.3em,
major grid style=thin,dotted,Gray,
point meta = explicit symbolic,
legend style =
at=(0.11,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em
legend image code/.code=
[#1,yshift=-0.3em] (0em,0em) rectangle (0.7em,0.7em);

Hit rate;
[draw=LineColorD,fill=ShadeColorD,text=LineColorD] coordinates
(32,    0.9503113414)[0.9503113414]
(64,    0.944796861 )[0.944796861 ]
(128,   0.938174762 )[0.938174762 ]
(256,   0.9306883521)[0.9306883521]
(512,   0.922097759 )[0.922097759 ]
(1024,  0.9123444323)[0.9123444323]
(2048,  0.9012861126)[0.9012861126]
(4096,  0.8887485274)[0.8887485274]
(8192,  0.8748170039)[0.8748170039]

width = , height = ,
axis x line = none,
xlabel = Batch size,
xtick = data, symbolic x coords =
32, 64, 128, 256, 512, 1024, 2048, 4096, 8192
xtick style = draw=none,
enlarge x limits = 0.075,
axis y line* = left,
ylabel style = align=center,
ylabel = Avg. Latency (  ),
ymin = 0, ymax = 100,
ytick = 25, 50, 75,
major tick length=0.3em,
legend style =
at=(0.62,1.015), anchor=south,
inner sep=0,
legend columns=-1,
/tikz/every even column/.append style=column sep=2em

color=LineColorA!50!White,very thick,
mark=*,mark options=LineColorA!90!White, scale=0.5
] coordinates
(32,      0.871078)[ 0.871078]
(64,      1.080958)[ 1.080958]
(128,     2.084895)[ 2.084895]
(256,     3.091692)[ 3.091692]
(512,     9.146828)[ 9.146828]
(1024,   13.048576)[13.048576]
(2048,   25.221714)[25.221714]
(4096,   42.71374 )[42.71374 ]
(8192,   88.425117)[88.425117]
color=LineColorF!70!White,very thick,
mark=*,mark options=LineColorF, scale=0.5
] coordinates
(32,      0.721225058)[  0.721225058]
(64,      0.891025355)[  0.891025355]
(128,     1.342232456)[  1.342232456]
(256,     2.385526316)[  2.385526316]
(512,     4.1079839  )[  4.1079839  ]
(1024,    7.404429032)[  7.404429032]
(2048,   14.94513031 )[ 14.94513031 ]
(4096,   27.31018078 )[ 27.31018078 ]
(8192,   53.99418906 )[ 53.99418906 ]
color=LineColorC!50!White,very thick,
mark=*,mark options=LineColorC!90!White, scale=0.5
] coordinates
(32,      0.6092552188)[  0.6092552188]
(64,      0.7407793728)[  0.7407793728]
(128,     1.034970386 )[  1.034970386 ]
(256,     1.590935848 )[  1.590935848 ]
(512,     2.802354975 )[  2.802354975 ]
(1024,    4.622808321 )[  4.622808321 ]
(2048,    8.633312714 )[  8.633312714 ]
(4096,   16.26296615  )[ 16.26296615  ]
(8192,   31.39529114  )[ 31.39529114  ]

Synthetic dataset B

Comparison of HPS performance with different NVIDIA GPUs.

§.§.§ Performance and accuracy comparison

To determine the influence of the hardware on HPS performance, we measure the stable stage inference latency with different datasets and batch sizes on respectively a NVIDIA T4 (16 GB memory), A30 (24 GB memory) and A100 GPU (80 GB memory). To allow for a fair comparison despite the limited memory compliment of the T4 GPU, we set the cache percentage and hit rate threshold to respectively 10% and 1.0 for all GPUs. Hence, for the same dataset, the different GPUs stabilize at the same hit rate throughout <ref>. For the comparatively tiny MovieLens (<ref>) dataset, we can achieve an inference latency of less than 1 ms with the hit rate saturating at ~98.5%. With Criteo 1 TB (<ref>) and Synthetic dataset B (<ref>), the hit rate gradually decreases as the batch size increases. Synthetic dataset B simulates a recommendation task where more categorical features are used. Larger batch sizes lead to increased computational overhead of the dense layers and higher overall inference latency. Thus, although their embedding tables are similarly sized, the latency figures for Synthetic dataset B are much higher than those for the Criteo 1 TB dataset.
Because large recommendation models often do not fit into GPU memory, most inference frameworks only load the dense model part into GPU memory. Meanwhile, HPS can deploy and accelerate such models on the GPU through its GPU cache mechanism. When using small and medium batch sizes with HPS, the inference latencies for T4 and A30 GPUs are on par with an A100 GPU, thus demonstrating that HPS is a scalable inference solution for recommender systems. To complete our investigation, we present the prediction accuracy, i.e., $\frac{\sum \text{correct}}{\sum \text{total}}$ samples, when varying the cache hit rate for the Criteo 1 TB dataset in <ref>. Here, the three different hit rate thresholds overlap almost perfectly, while their cache hit rates stabilize above 0.9. This strongly implies that the GPU embedding cache retains hot embeddings well, even if the asynchronous insertion is employed.

§.§ Online update performance

Our updating mechanism consists of three major components (cf. <ref>): (1) dumping the model in the training nodes, (2) update ingestion by the inference nodes, and (3) the embedding cache refresh operation. Because model dumps are done in isolation from the overall HPS inference system, we only focus on the remaining two components.

Update ingestion mechanism. The delivery ratio of model parameter updates depends heavily on the configuration of the intermediate Kafka message buffer storage and its network connectivity. Of the latter, we have plenty (cf. <ref>). Therefore, we concentrate on the receiving VDB/PDB instances. In <ref>, we perform asynchronous random batch insertion (batch size = 128 MB) limits for Synthetic dataset A in our test environment. The insertion speed slowly declines as the model size increases due to storage management overheads.

GPU embedding cache refresh. In <ref>, we analyze the embedding cache refresh performance with different cache capacities, and show the impact of latency and capacity. Note how the overhead for actually dumping the embedding keys scales with the cache size, but is almost negligible in comparison to the following update operation. The throughput remains stable around 199 GB/s, so that it only takes 200 ms to refresh a 40 GB embedding cache.

To summarize, online updates only have a minor impact on the overall inference performance. That is because the VDB/PDB update operations happen lazily and infrequently, and subsequent cache refreshes happen near-instantaneously.

§ CONCLUSION

In this paper we presented and analyzed HPS, an efficient GPU-enabled hierarchical parameter server for building large-scale model inference services. Our high-performance GPU embedding cache exploits the typical properties of recommendation datasets to improve inference throughput. By extending this GPU embedding cache with other cluster storage resources (VDB & PDB), HPS can efficiently process queries for very large models. Through its asynchronous update mechanisms, HPS ensures that its GPU embedding cache retains a high hit rate over time.

Our experiments show that the HugeCTR HPS can reduce the latency for end-to-end model inference by 5-62x in comparison with PyTorch CPU. Furthermore, HPS offers excellent scaling and performance on different GPUs.

As for future work, we intend to continue extending Merlin HugeCTR and HPS with additional features, including but not limited to better support for next generation GPU technologies and further optimizing performance, like relaxing the constraints when using locks to protect data and ensure thread-safety for embedding cache.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Arefyeva et al(2018)]
authorpersonIya Arefyeva, personDavid
Broneske, personGabriel Campero, personMarcus
Pinnecke, and personGunter Saake.
Memory Management Strategies in CPU/GPU Database
Systems: A Survey. In booktitleBeyond Databases,
Architectures and Structures. Facing the Challenges of Data Proliferation and
Growing Variety. publisherSpringer International Publishing,
addressCham, pages128–142.

[Ben-Nun and Hoefler(2019)]
authorpersonTal Ben-Nun and personTorsten
Hoefler. year2019.
Demystifying Parallel and Distributed Deep
Learning: An In-Depth Concurrency Analysis.
journalACM Comput. Surv. volume52,
number4, Article articleno65 (dateaug
year2019), numpages43 pages.

[Clauset et al(2009)]
authorpersonAaron Clauset,
personCosma Rohilla Shalizi, and personM. E. J.
Newman. year2009.
Power-Law Distributions in Empirical Data.
journalSIAM Rev. volume51,
number4 (year2009), pages661–703.

authorpersonYann Collet.
titlexxHash Hash Function.
Accessed: 2022-04-15.

[Crankshaw et al(2017)]
authorpersonDaniel Crankshaw, personXin
Wang, personGuilio Zhou, personMichael J. Franklin,
personJoseph E. Gonzalez, and personIon Stoica.
Clipper: A Low-Latency Online Prediction Serving
System. In booktitle14th USENIX Symposium on Networked
Systems Design and Implementation (NSDI 17). publisherUSENIX
Association, addressBoston, MA, pages613–627.

[Criteo AI Lab(2014)]
authorpersonCriteo AI Lab.
titleCriteo 1TB Click Logs dataset.
Accessed: 2022-03-15.

[Cui et al(2016)]
authorpersonHenggang Cui, personHao Zhang,
personGregory R. Ganger, personPhillip B. Gibbons,
and personEric P. Xing. year2016.
GeePS: Scalable Deep Learning on Distributed GPUs
with a GPU-Specialized Parameter Server. In
booktitleProceedings of the 11th European Conference on
Computer Systems (London, United Kingdom) (seriesEuroSys
'16). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, Article articleno4,
numpages16 pages.

[Farrell et al(2021)]
authorpersonSteven Farrell, personMurali
Emani, personJacob Balma, personLukas Drescher,
personAleksandr Drozd, personAndreas Fink,
personGeoffrey Fox, personDavid Kanter,
personThorsten Kurth, personPeter Mattson,
personDawei Mu, personAmit Ruhela,
personKento Sato, personKoichi Shirahata,
personTsuguchika Tabaru, personAristeidis Tsaris,
personJan Balewski, personBen Cumming,
personTakumi Danjo, personJens Domke,
personTakaaki Fukai, personNaoto Fukumoto,
personTatsuya Fukushi, personBalazs Gerofi,
personTakumi Honda, personToshiyuki Imamura,
personAkihiko Kasagi, personKentaro Kawakami,
personShuhei Kudo, personAkiyoshi Kuroda,
personMaxime Martinasso, personSatoshi Matsuoka,
personHenrique Mendonça, personKazuki Minami,
personPrabhat Ram, personTakashi Sawada,
personMallikarjun Shankar, personTom St. John,
personAkihiro Tabuchi, personVenkatram Vishwanath,
personMohamed Wahib, personMasafumi Yamazaki, and
personJunqi Yin. year2021.
MLPerf™ HPC: A Holistic Benchmark
Suite for Scientific Machine Learning on HPC Systems. In
booktitleIEEE/ACM Workshop on Machine Learning in High
Performance Computing Environments (MLHPC) (St. Louis, MO, USA).
publisherIEEE Press, addressNew York, NY, USA,

[Goodwin et al(2021a)]
authorpersonDavid Goodwin et al
titleNVIDIA Triton: Model Control Mode.
Accessed: 2022-07-19.

[Goodwin et al(2021b)]
authorpersonDavid Goodwin et al
titleTriton Architecture - Concurrent Model Execution.
Accessed: 2022-07-19.

[Goodwin et al(2022)]
authorpersonDavid Goodwin et al
titleNVIDIA Triton: Performance Analyzer.
Accessed: 2022-07-19.

[Guo et al(2021)]
authorpersonHuifeng Guo, personWei Guo,
personYong Gao, personRuiming Tang,
personXiuqiang He, and personWenzhi Liu.
booktitleScaleFreeCTR: MixCache-Based Distributed
Training System for CTR Models with Huge Embedding Table.
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages1269–1278.

[Gupta et al(2020)]
authorpersonUdit Gupta, personSamuel Hsia,
personVikram Saraph, personXiaodong Wang,
personBrandon Reagen, personGu-Yeon Wei,
personHsien-Hsin S. Lee, personDavid Brooks, and
personCarole-Jean Wu. year2020.
DeepRecSys: A System for Optimizing End-To-End
At-Scale Neural Recommendation Inference. In
booktitleACM/IEEE 47th Annual International Symposium on
Computer Architecture (ISCA). publisherIEEE Press,
addressValencia, Spain, pages982–995.

[Gupta et al(2021)]
authorpersonVipul Gupta, personDhruv
Choudhary, personPeter Tang, personXiaohan Wei,
personXing Wang, personYuzhen Huang,
personArun Kejariwal, personKannan Ramchandran, and
personMichael W. Mahoney. year2021.
Training Recommender Systems at Scale:
Communication-Efficient Model and Data Parallelism. In
booktitleProceedings of the 27th ACM SIGKDD Conference on
Knowledge Discovery & Data Mining (Virtual Event, Singapore)
(seriesKDD '21). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Harper and Konstan(2015)]
authorpersonF. Maxwell Harper and
personJoseph A. Konstan. year2015.
The MovieLens Datasets: History and Context.
journalACM Trans. Interact. Intell. Syst.
volume5, number4, Article articleno19
(year2015), numpages19 pages.

[Huang et al(2018)]
authorpersonYuzhen Huang, personTatiana
Jin, personYidi Wu, personZhenkun Cai,
personXiao Yan, personFan Yang,
personJinfeng Li, personYuying Guo, and
personJames Cheng. year2018.
FlexPS: Flexible Parallelism Control in Parameter
Server Architecture.
journalProc. VLDB Endow. volume11,
number5 (year2018), pages566–579.

[Jiang et al(2017)]
authorpersonJie Jiang, personLele Yu,
personJiawei Jiang, personYuhong Liu, and
personBin Cui. year2017.
Angel: a new large-scale machine learning system.
journalNational Science Review
volume5, number2 (year2017),

[Jiang et al(2021)]
authorpersonWenqi Jiang, personZhenhao He,
personShuai Zhang, personKai Zeng,
personLiang Feng, personJiansong Zhang,
personTongxuan Liu, personYong Li,
personJingren Zhou, personCe Zhang, and
personGustavo Alonso. year2021.
FleetRec: Large-Scale Recommendation Inference on
Hybrid GPU-FPGA Clusters. In booktitleProceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual
Event, Singapore) (seriesKDD '21).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages3097–3105.

[Kanter et al(2021)]
authorpersonDavid Kanter, personPeter
Mattson, et al year2021.
titleML$\cdot$Commons / MLperf v1.1 Results.
Accessed: 2022-03-15.

[Langer et al(2020)]
authorpersonMatthias Langer, personZhen He,
personYanbo Xue, and personWenny Rahayu.
Distributed Training of Deep Learning Models: A
Taxonomic Perspective.
journalIEEE Transactions on Parallel and Distributed
Systems volume31, number12 (year2020),

[Lui et al(2021)]
authorpersonMichael Lui, personYavuz Yetim,
personÖzgür Özkan, personZhuoran
Zhao, personShin-Yeh Tsai, personCarole-Jean Wu,
and personMark Hempstead. year2021.
Understanding Capacity-Driven Scale-Out Neural
Recommendation Inference. In booktitle2021 IEEE
International Symposium on Performance Analysis of Systems and Software
(ISPASS) (Stony Brook, NY, USA). publisherIEEE Press,
addressNew York, NY, USA, pages162–171.

[Luo et al(2018)]
authorpersonLiang Luo, personJacob Nelson,
personLuis Ceze, personAmar Phanishayee, and
personArvind Krishnamurthy. year2018.
Parameter Hub: A Rack-Scale Parameter Server for
Distributed Deep Neural Network Training. In
booktitleProceedings of the ACM Symposium on Cloud
Computing (Carlsbad, CA, USA) (seriesSoCC '18).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages41–54.

[Mittal and Vetter(2015)]
authorpersonSparsh Mittal and
personJeffrey S. Vetter. year2015.
A Survey of CPU-GPU Heterogeneous Computing
journalACM Comput. Surv. volume47,
number4, Article articleno69 (year2015),
numpages35 pages.

[Naumov et al(2019)]
authorpersonMaxim Naumov, personDheevatsa
Mudigere, personHao-Jun Michael Shi, personJianyu
Huang, personNarayanan Sundaraman, personJongsoo
Park, personXiaodong Wang, personUdit Gupta,
personCarole-Jean Wu, personAlisson G. Azzolini,
personDmytro Dzhulgakov, personAndrey Mallevich,
personIlia Cherniavskii, personYinghai Lu,
personRaghuraman Krishnamoorthi, personAnsha Yu,
personVolodymyr Kondratenko, personStephanie Pereira,
personXianjie Chen, personWenlin Chen,
personVijay Rao, personBill Jia,
personLiang Xiong, and personMisha Smelyanskiy.
Deep Learning Recommendation Model for
Personalization and Recommendation Systems.
journalCoRR  volumeabs/1906.00091
(year2019), numpages10 pages.

authorpersonNVIDA. year2022a.
titleHugeCTR: Distributed deployment with the
Hierarchical Parameter Server.
Accessed: 2022-04-15.

authorpersonNVIDA. year2022b.
titleHugeCTR Hierarchical Parameter Server.
Accessed: 2022-04-15.

authorpersonNVIDIA. year2020.
titleNVIDIA A100 Tensor Core GPU.
, numpages82 pages.

authorpersonNVIDIA. year2022a.
titleCUDA C Programming Guide.
, numpages131 pages.
Accessed: 2022-04-15.

authorpersonNVIDIA. year2022b.
titleHugeCTR: Embedding Cache Asynchronous Insertion
Accessed: 2022-04-15.

authorpersonNVIDIA. year2022c.
titleHugeCTR: GPU Cache.
Accessed: 2022-04-15.

authorpersonNVIDIA. year2022d.
titleHugeCTR: Triton Backend.
Accessed: 2022-05-15.

[NVidia Corp.(2022)]
authorpersonNVidia Corp.
titleSolutions for the Data-Center.
Accessed: 2022-04-15.

[NVIDIA Deep Learning Examples for Tensor Cores(2022a)]
authorpersonNVIDIA Deep Learning Examples for Tensor
Cores. year2022a.
titleDeploying the DLRM model using Triton Inference

[NVIDIA Deep Learning Examples for Tensor Cores(2022b)]
authorpersonNVIDIA Deep Learning Examples for Tensor
Cores. year2022b.
titleDLRM For TensorFlow 2.

[Rosenfeld et al(2022)]
authorpersonViktor Rosenfeld,
personSebastian Breß, and personVolker Markl.
Query Processing on Heterogeneous CPU/GPU
journalACM Comput. Surv. volume55,
number1, Article articleno11 (year2022),
numpages38 pages.

authorpersonMatthias J. Sax.
booktitleApache Kafka.
publisherSpringer International Publishing,
addressCham, pages1–8.

[Subramanian et al(2021)]
authorpersonHarish Kumar Harihara Subramanian,
personBala Gurumurthy, personGabriel Campero Durand,
personDavid Broneske, and personGunter Saake.
Analysis of GPU-Libraries for Rapid Prototyping
Database Operations: A look into library support for database operations.
journalProceedings of the 37th International
Conference on Data Engineering Workshops (ICDEW)  volume1
(year2021), pages36–41.

[Sun et al(2019)]
authorpersonFei Sun, personJun Liu,
personJian Wu, personChanghua Pei,
personXiao Lin, personWenwu Ou, and
personPeng Jiang. year2019.
BERT4Rec: Sequential Recommendation with
Bidirectional Encoder Representations from Transformer. In
booktitleProceedings of the 28th ACM International
Conference on Information and Knowledge Management (Beijing, China)
(seriesCIKM '19). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Vaswani et al(2017)]
authorpersonAshish Vaswani, personNoam
Shazeer, personNiki Parmar, personJakob Uszkoreit,
personLlion Jones, personAidan N. Gomez,
personŁukasz Kaiser, and personIllia
Polosukhin. year2017.
Attention is All You Need. In
booktitleProceedings of the 31st International Conference on
Neural Information Processing Systems (Long Beach, California, USA)
(seriesNIPS'17). publisherCurran Associates
Inc., addressRed Hook, NY, USA, pages6000–6010.

[Wilkening et al(2021)]
authorpersonMark Wilkening, personUdit
Gupta, personSamuel Hsia, personCaroline Trippel,
personCarole-Jean Wu, personDavid Brooks, and
personGu-Yeon Wei. year2021.
RecSSD: Near Data Processing for Solid State Drive
Based Recommendation Inference. In booktitleProceedings of
the 26th ACM International Conference on Architectural Support for
Programming Languages and Operating Systems (Virtual, USA)
(seriesASPLOS 2021). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Yuan et al(2013)]
authorpersonYuan Yuan, personRubao Lee,
and personXiaodong Zhang. year2013.
The Yin and Yang of Processing Data Warehousing
Queries on GPU Devices.
journalProc. VLDB Endow. volume6,
number10 (year2013), pages817–828.

[Zhao et al(2020)]
authorpersonWeijie Zhao, personDeping Xie,
personRonglai Jia, personYulei Qian,
personRuiquan Ding, personMingming Sun, and
personPing Li. year2020.
Distributed Hierarchical GPU Parameter Server for
Massive Scale Deep Learning Ads Systems.
journalProceedings of Machine Learning and Systems
volume2 (year2020), pages412–428.