#  _DaeMon_ : Architectural Support for Efficient Data Movement in
Disaggregated Systems

Christina Giannoula<EMAIL_ADDRESS>University of TorontoCanada
National Technical University of AthensGreece ,  Kailong Huang  University of
TorontoCanada ,  Jonathan Tang  University of TorontoCanada ,  Nectarios
Koziris  National Technical University of AthensGreece ,  Georgios Goumas
National Technical University of AthensGreece ,  Zeshan Chishti  Intel
CorporationUSA and  Nandita Vijaykumar  University of TorontoCanada

(2023)

###### Abstract.

Resource disaggregation offers a cost effective solution to resource scaling,
utilization, and failure-handling in data centers by physically separating
hardware devices in a server. Servers are architected as pools of processor,
memory, and storage devices, organized as independent failure-isolated
components interconnected by a high-bandwidth network. A critical challenge,
however, is the high performance penalty of accessing data from a remote
memory module over the network. Addressing this challenge is difficult as
disaggregated systems have high runtime variability in network
latencies/bandwidth, and page migration can significantly delay critical path
cache line accesses in other pages.

This paper conducts a characterization analysis on different data movement
strategies in fully disaggregated systems, evaluates their performance
overheads in a variety of workloads, and  introduces _DaeMon_ , the first
software-transparent mechanism to significantly alleviate data movement
overheads in fully disaggregated systems. First, to enable scalability to
multiple hardware components in the system, we enhance each compute and memory
unit with specialized engines that transparently handle data migrations.
Second, to achieve high performance and provide robustness across various
network, architecture and application characteristics, we implement a
synergistic approach of bandwidth partitioning, link compression, decoupled
data movement of multiple granularities, and adaptive granularity selection in
data movements. We evaluate _DaeMon_ in a wide variety of workloads at
different network and architecture configurations using a state-of-the-art
simulator. _DaeMon_ improves system performance and data access costs by
2.39$\times$ and 3.06$\times$, respectively, over the widely-adopted approach
of moving data at page granularity.

data movement, data access, memory access, hardware support, hardware
mechanism, high performance, memory systems, memory disaggregation, resource
disaggregation, disaggregated systems, workload characterization,
benchmarking, performance characterization

††copyright: acmcopyright††journalyear: 2023††journal: POMACS††journalvolume:
7††journalnumber: 1††publicationmonth: 3

##  1\. Introduction

With recent advances in network technologies (Mellanox, 2020; RDMA, 2019;
Intel, 2021; RDMA, 2022; Gao et al., 2016) that enable high bandwidth
networks, resource disaggregation (Shan et al., 2018; Gao et al., 2016) has
emerged as a promising technology for data centers (Shan et al., 2018; Lee et
al., 2021; Calciu et al., 2021; Gao et al., 2016; Tsai and Zhang, 2017; Guo et
al., 2022; Giannoula, 2022). Resource disaggregation proposes the physical
separation of hardware devices (CPU, accelerator, memory, and disk) in a
server as independent and failure-isolated components connected over a high-
bandwidth network such as RDMA (Mellanox, 2020) and Gen-Z (RDMA, 2022).
Compared to monolithic servers that tightly integrate these components (Figure
1a), _disaggregated systems_ can greatly improve _resource utilization_ , as
memory/storage components can be shared across applications; _resource
scaling_ , as hardware components can be flexibly added, removed, or upgraded;
and _failure handling_ , as the entire server does not need to be replaced in
the event of a fault in a device. Thus, resource disaggregation can
significantly decrease data center costs.

Disaggregated systems comprise multiple compute, memory and storage
components, interconnected over a high-bandwidth network (Figure 1b), each
independently managed by a specialized kernel module (monitor). Typically,
each _compute component_ includes a small amount (a few GBs) of main memory
(henceforth referred to as _local memory_) to improve memory performance.
However, almost all the memory in the data center is separated as network-
attached disaggregated memory components to maximize resource sharing and
independence in failure handling (different from typical hybrid memory
architectures). Thus, the _majority_ of the application working sets is
accessed from the disaggregated memory components (henceforth referred to as
_remote memory_). Each _memory component_ includes its own controller and can
be flexibly shared by many compute components. Thus, disaggregated systems can
provide high memory capacity for applications with large working sets (e.g.,
bioinformatics, graph processing and neural networks) at lower cost. Fine-
grain microsecond-latency networking technologies (Mellanox, 2020; RDMA, 2019;
Intel, 2021; RDMA, 2022; Guo et al., 2016) that interconnect all hardware
components have made fully disaggregated systems feasible, being only
2-8$\times$ slower than DRAM bus bandwidth. However, since a large fraction of
the application’s data (typically $\sim$80%) (Shan et al., 2018; Lee et al.,
2021; Gao et al., 2016) is located and accessed from remote memory, the higher
latencies of remotely accessing data over the network can cause large
performance penalties.

Figure 1. (a) A monolithic system versus (b) a disaggregated system.

Alleviating data access overheads is challenging in disaggregated systems for
the following reasons. First, disaggregated systems are not monolithic and
comprise independently managed entities: each component has its own hardware
controller, its resource allocation is transparent from other components and a
specialized kernel monitor uses its own functionality/implementation to manage
the component it runs on (only communicates with other monitors via network
messaging if there is a need to access remote resources). This characteristic
necessitates a distributed and disaggregated solution that can scale to a
large number of independent components in the system. Second, there is high
variability in data access latencies as they depend on the location of the
remote memory component and the contention with other compute components that
share the same memory components and network. Data placements can also vary
during runtime or between multiple executions, since data is dynamically
allocated in one or more remote memory components and hardware updates can
flexibly change the architecture of the memory component and the network
topology. Third, data is typically migrated at page granularity (Yan et al.,
2019; Aguilera et al., 2017; Zhang et al., [n. d.]; Lim et al., 2012; Shan et
al., 2018; Angel et al., 2020; Gu et al., 2017; Aguilera et al., 2018; Lee et
al., 2021) as it enables: (i) transparency to avoid modifications to existing
OS/applications; (ii) low metadata overheads for address translation; and
(iii) leveraging spatial locality within pages. However, we observe in §2.2
that moving memory pages in disaggregated systems, i.e., moving data at a
large granularity over the network, can significantly increase bandwidth
consumption and slow down accesses to cache lines in other concurrently
accessed pages.

Recent works on hybrid memory systems (Dong et al., 2010; Doudali et al.,
2019; Yan et al., 2019; Liu et al., 2017; Kotra et al., 2018; Agarwal and
Wenisch, 2017; Meswani et al., 2015; Ruan et al., 2020; Dulloor et al., 2016;
Agarwal et al., 2015; Kim et al., 2021; Doudali et al., 2021; Liu et al.,
2019; Kannan et al., 2017; Loh and Hill, 2011, 2012; Chou et al., 2014; Ryoo
et al., 2017; Jevdjic et al., 2013), for example, those that integrate die-
stacked DRAM (Jun et al., 2017) caches aim to address the high page movement
costs between main memory and the DRAM cache (Loh and Hill, 2011, 2012; Chou
et al., 2014; Ryoo et al., 2017; Jevdjic et al., 2013) with mechanisms to move
data at smaller granularities (Vasilakis et al., 2019; Jiang et al., 2010;
Prodromou et al., 2017; Loh and Hill, 2011, 2012; Chou et al., 2014;
Vasilakis, Evangelos and Papaefstathiou, Vassilis and Trancoso, Pedro and
Sourdis, Ioannis, 2020), e.g., cache line, or by using page placement/hot page
selection mechanisms (Dong et al., 2010; Doudali et al., 2019; Yan et al.,
2019; Liu et al., 2017; Kotra et al., 2018; Agarwal and Wenisch, 2017; Meswani
et al., 2015; Ruan et al., 2020; Dulloor et al., 2016; Agarwal et al., 2015;
Kim et al., 2021; Doudali et al., 2021; Liu et al., 2019; Kannan et al.,
2017). However, these prior works are tailored for a monolithic tightly-
integrated architecture (Figure 1a), and are not suitable for disaggregated
systems (See § 7). These works assume _centralized_ data management/allocation
(unlike in disaggregated systems). For instance, software runtimes (Dong et
al., 2010; Doudali et al., 2019; Yan et al., 2019; Liu et al., 2017; Kotra et
al., 2018; Agarwal and Wenisch, 2017; Meswani et al., 2015; Ruan et al., 2020;
Dulloor et al., 2016; Agarwal et al., 2015; Kim et al., 2021; Doudali et al.,
2021; Liu et al., 2019; Kannan et al., 2017) running on CPUs in hybrid systems
leverage TLBs/page tables to track page hotness and move pages across
different memory devices (Figure 1a). Instead, in fully disaggregated systems
all hardware memory functionalities (e.g., TLBs, page tables) of remote pages
are moved to the memory components themselves (Shan et al., 2018; Guo et al.,
2022) (Figure 1b). Thus they cannot be used to track page hotness at the CPU
side to implement intelligent page placement/movement in local memory.
Similarly, hardware-based approaches (Vasilakis et al., 2019; Jiang et al.,
2010; Prodromou et al., 2017; Li et al., 2017) add _centralized_ hardware
units in the CPU to track metadata for pages in second-tier memory. This
however would incur high hardware costs in disaggregated systems that enable
large amounts (e.g., TBs) of remote memory (Shan et al., 2018). Requiring
_each_ compute component to control/track a large number of pages in remote
memory components would impose significant hardware costs and scalability
challenges, and thus might annihilate the benefits of resource disaggregation.
Moreover, disaggregated systems incur significant variations in access
latencies and bandwidth based on the current network architecture and
concurrent jobs sharing the memory components/network, which are not addressed
by prior work. This necessitates a solution primarily designed for robustness
to this variability.

In this work, we analyze different data movement strategies in fully
disaggregated systems, and introduce _DaeMon_ , an efficient software-
transparent mechanism to alleviate data movement overheads in disaggregated
systems. _DaeMon_ provides (i) high performance on dynamic workload demands,
(ii) robustness to variations in architectures, network characteristics and
application behavior, and (iii) independence and scalability to multiple
compute components and memory components that are managed transparently to
each other and are flexibly added/removed in the system.

_DaeMon_ consists of two key ideas. First, we offload data migrations to
dedicated hardware engines, named _DaeMon_ compute and memory engine, that are
added at each compute component and memory component, respectively. This key
idea enables independence and scalability to a large number of compute
components and memory components of disaggregated systems. Compared to a
centralized design, _DaeMon_ ’s distributed management of data movement
enables simultaneous processing of data movement across multiple components
and decreases the processing costs and queuing delays to serve data requests.
Second, we leverage the synergy of three key techniques to provide robustness
to the high variability in network latencies/bandwidth. 1) We use a bandwidth
partitioning approach to enable the decoupled movement of data at _two
granularities_ , i.e., page and cache line, and _prioritize_ cache line
granularity data moves over page moves. This design enables low access
latencies to remote memory for the cache line requests on the critical path,
while the associated pages can be still be moved independently at slower rates
to retain the benefits of spatial locality. 2) We design an adaptive approach
to decide on-the-fly if a request should be served by a cache line, page or
_both_. Via selective granularity data movement, we provide robustness to
variations in network, architectures and application characteristics. 3) We
leverage hardware link compression when migrating pages to reduce network
bandwidth consumption and alleviate queuing delays.

The synergy of the aforementioned key techniques provides a robust solution
for disaggregated systems: decoupled multiple granularity data movement
effectively prioritizes cache line requests on the critical path, and migrates
pages at a slower rate leveraging compression to reduce bandwidth consumption.
The adaptive granularity selection mechanism effectively adapts to the
characteristics of the application data, e.g., by favoring moving more pages
if application data is highly compressible. The decoupled cache line
granularity movement also enables the use of more sophisticated and effective
compression algorithms (with relatively high compression latency) for page
migrations.

We evaluate _DaeMon_ using a range of capacity intensive workloads with
different memory access patterns from machine learning, high-performance-
computing, graph processing, and bioinformatics domains. Over the widely-
adopted approach of moving data at page granularity, _DaeMon_ decreases memory
access latencies by 3.06$\times$ on average, and improves system performance
by 2.39$\times$ on average. We demonstrate that _DaeMon_ provides (i)
robustness and significant performance benefits on various
network/architecture configurations and application behavior (Figures 8 and
13), (ii) scalability to _multiple_ hardware components and networks, (Figure
17), and (iii) adaptivity to dynamic workload demands, even when multiple
heterogeneous jobs are concurrently executed in the disaggregated system
(Figure 18).

This paper makes the following contributions:

  * •

We heavily modify a state-of-the-art simulator to develop and evaluate the
overheads of different data movement strategies in fully disaggregated
systems, analyze the challenges of providing efficient data movement in such
systems, and develop _DaeMon_ , an adaptive distributed data movement
mechanism for fully disaggregated systems.

  * •

We enable decoupled data movement at two granularities, and migrate the
requested critical data quickly at cache line granularity and the
corresponding pages opportunistically without stalling critical cache line
requests. We dynamically control the data movement granularity to effectively
adapt to the current system load and application behavior. We employ a high-
latency compression scheme to further reduce bandwidth consumption during page
migrations.

  * •

We evaluate _DaeMon_ using a wide range of capacity intensive workloads,
various architecture/network configurations, and in multi-workload executions
of concurrent heterogeneous jobs. We demonstrate that _DaeMon_ significantly
outperforms the state-of-the-art data movement strategy, and constitutes a
robust and scalable approach for data movement in fully disaggregated systems.

##  2\. Background and Motivation

###  2.1. Baseline Disaggregated System

Figure 2 shows the baseline organization of the disaggregated system, which
includes several compute components and memory components as network-attached
components. To improve performance, each compute component tightly includes a
few GBs of main memory, referred to as local memory, which can typically host
20-25% of the application’s memory footprint (Shan et al., 2018; Gao et al.,
2016; Sapio et al., 2017). Each memory component includes its own controller
and connects multiple DIMM modules, referred to as remote memory.

Figure 2. High-level organization of a disaggregated system.

We assume distributed OS modules that coordinate and communicate with each
other via network messaging when needed, similar to (Shan et al., 2018; Lee et
al., 2021; Guo et al., 2022): processor and memory kernel monitors run at
compute components and memory components, respectively. The memory
allocation/management of remote memory is performed at the memory component
itself (Guo et al., 2022; Shan et al., 2018), transparently to compute
components, enabling the different components to be _independent_. The on-chip
caches and the local memory of compute components are typically indexed by
virtual addresses (Shan et al., 2018), and remote data is requested from
memory components using virtual addresses (Shan et al., 2018; Lee et al.,
2021; Calciu et al., 2021; Guo et al., 2022) (unlike in hybrid memory
systems). The data management is typically performed at page granularity
(Calciu et al., 2021; Shan et al., 2018; Lee et al., 2021) (e.g., 4KB). The
local memory of the compute component can be treated as a cache with tags
(Shan et al., 2018) or a _local_ virtual to physical translation mapping (Lee
et al., 2021) can be used (either approach works with _DaeMon_ , however we
assume the second approach in our evaluation). The physical memory addresses
of the local memory can be found by accessing and traversing metadata (tags or
local page tables) kept in a dedicated (pre-reserved) DRAM memory space
(kernel metadata is directly indexed via physical addresses). When an local
memory _miss_ happens, either (i) the processor kernel module of the compute
component triggers a page fault and fetches the requested page from remote
memory components (Shan et al., 2018), or (ii) dedicated software runtimes co-
designed with hardware primitives (Calciu et al., 2021) (e.g., supported in
FPGA-based controllers as shown in Figure 2a) handle remote data requests on
demand completely eliminating expensive page faults. Either approach works
with _DaeMon_. We assume that the controllers of memory components implement
_hardware-based_ address translation (Figure 2b) to access pages in remote
memory as proposed in (Guo et al., 2022).

Jobs running at different compute components can share _read-only_ pages
located at multiple memory components. Similarly to prior state-of-the-art
works (Shan et al., 2018; Guo et al., 2022; Calciu et al., 2021), we assume
that the system does not support writable shared pages across compute
components, since they are rare across datacenter jobs (Shan et al., 2018; Guo
et al., 2022).

###  2.2. Data Movement Overheads in Fully Disaggregated Systems

Prior state-of-the-art works (Wang et al., 2020; Zuo et al., 2021; Zhang et
al., [n. d.]; Bindschaedler et al., 2020; Peng et al., 2020; Pinto et al.,
2020; Katrinis et al., 2016; Guo et al., 2022; Gu et al., 2017; Lee et al.,
2021) typically enable data management at page granularity for three
compelling reasons. First, the memory allocation and management is
transparent, i.e., requires little to no modification to OS or application
code. Second, the coarse granularity enables low metadata overheads for
address translation in local memory and remote memory. Managing local memory
as a cache at cache line granularity would incur prohibitively high metadata
overheads (Shan et al., 2018). Third, page movements enable exploiting
_spatial locality_ in common memory access patterns (Lim et al., 2009; Jiang
et al., 2010; Woo et al., 2010), and increase the number of accesses served
from the lower cost local memory instead of remote memory.

Figure 3 compares the performance of different data movement strategies in
disaggregated systems across various workloads (See Table 3). We evaluate one
memory component and one compute component having local memory to fit
$\sim$20% of the application’s working set. We use 100 ns/400 ns latency (Gao
et al., 2016; Lee et al., 2021) to model the propagation and switching delays
on the network (referred to as _switch latency_), and configure the network
bandwidth between the compute component and the memory component (referred to
as _bandwidth factor_) to be 1/4$\times$ the DRAM bus bandwidth (Shan et al.,
2018; Gao et al., 2016) of the local memory or remote memory. We compare six
configurations: (i) Local: all accesses are served from the local memory; (ii)
_cache-line_ : accessing data from remote memory at cache line granularity,
and directly writing data to the Last Level Cache (LLC) of the compute
component (local memory is not used), (iii) Remote: accessing data from remote
memory at page granularity (moving pages to local memory) accounting for all
network-related overheads, (iv) _page-free_ : remote accesses incur the
latency of one cache line granularity remote access and the corresponding page
is transferred to local memory at _zero cost_ (spatial locality is leveraged),
(v) _cache-line+page_ : requesting data from remote memory at both cache line
(moved to LLC) and page granularity (moved to local memory) and servicing data
requests using the latency of the packet that arrives earlier to compute
component (accounting for all network-related overheads), and (vi) __DaeMon__
: accessing data from remote memory using _DaeMon_ (accounting for all
network-related overheads).

Figure 3. Data movement overheads in disaggregated systems.

We make four observations. First, Remote, i.e., the typically-used approach of
moving data at page granularity, incurs significant performance slowdowns, on
average 3.86$\times$, over the monolithic Local configuration due to
transferring large amounts of data over the network. In addition to the large
network bandwidth consumption, migrating pages can slow down critical path
accesses to data in other concurrently accessed pages. Second, _page-free_
achieves almost the same performance as the Local scheme. A small penalty is
incurred as the first access to a page in remote memory incurs cache line
granularity latency access cost. However, since the whole page is migrated to
local memory _for free_ , performance significantly improves thanks to spatial
locality benefits of migrating pages in addition to the requested cache line.
Thus, migrating pages to local memory is critical to achieving high
performance. Third, _cache-line_ outperforms Remote in some latency-bound
workloads with poor spatial locality, however its performance benefits depend
on network characteristics. For example, in _tr_ , _cache-line_ outperforms
Remote by 1.42$\times$ when the switch latency is 100 ns, while it incurs
1.82$\times$ performance slowdown over Remote with 400 ns switch latency.
Fourth, the _cache-line+page_ scheme, i.e., _naively_ moving data at _both_
granularities, is still inefficient (only 1.11$\times$ better than Remote),
since critical cache lines are still queued behind large pages.

Overall, we draw two conclusions. (i) Page migrations incur high performance
penalties and can significantly slow down the critical path cache line
requests to other concurrently accessed pages. However, if the overheads of
migrating pages can be mitigated, moving data at page granularity offers a
critical opportunity to alleviate remote access costs. (ii) There is no _one-
size-fits-all_ granularity in data movements to always perform best across all
network configurations and applications. Depending on the spatial locality and
the network load, some applications benefit from cache line-only accesses that
avoid unnecessary congestion of pages in the network, while some applications
significantly benefit from page movements that leverage spatial locality. To
this end, we design _DaeMon_ to significantly reduce data movement costs
across various application, network and architecture characteristics. Figure 3
demonstrates that _DaeMon_ significantly outperforms the Remote and _cache-
line+page_ schemes by on average 2.38$\times$ and 2.14$\times$, respectively.

##  3\. _DaeMon_ : Our Approach

_DaeMon_ is an adaptive and scalable data movement mechanism for fully
disaggregated systems that supports low-overhead page migration, enables
software transparency, and provides robustness to variations in memory
component placements, network architectures and application behavior. _DaeMon_
comprises two key ideas:

(1) Disaggregated Hardware Support for Data Movement Acceleration. We enhance
each compute component and memory component with specialized engines, i.e.,
_DaeMon_ compute and memory engine (Figure 4), respectively, to manage data
movements across the network of disaggregated systems. _DaeMon_ engines enable
independence and high scalability to a large number of compute
components/memory components that are flexibly added/removed in disaggregated
systems. Moreover, distributed management of data migrations at multiple
_DaeMon_ engines increases the execution parallelism and decreases the
processing costs and queuing delays to serve data requests.

(2) Synergy of Three Key Techniques. _DaeMon_ incorporates three synergistic
key techniques shown in Figure 4:

Figure 4. High-level overview of _DaeMon_.

(I) _Decoupled Multiple Granularity Data Movement_. First, we integrate two
separate hardware queues to manage and serve data requests from remote memory
at two granularities, i.e., cache line (via the sub-block queue) and page (via
the page queue) granularity. Cache line requests are directly moved to Last
Level Cache (LLC) of the compute component to avoid additional metadata
overheads and eliminate memory latency. Page requests are moved to local
memory of the compute component. Second, we _prioritize_ moving cache lines
over moving pages via a bandwidth partitioning approach: a queue controller
serves cache line and page requests with a _predefined fixed ratio_ to ensure
that at any given time a certain fraction of the bandwidth resources is
_always allocated_ to serve cache line requests quickly. _DaeMon_ implements
both network and remote memory bus bandwidth partitioning.

This technique provides two benefits. First, retaining page migrations in
_DaeMon_ (i) enables software-transparency, (ii) allows maintaining metadata
for DRAM at page granularity (thus incurring low metadata overheads), and
(iii) exploits the performance benefits of data (spatial) locality within
pages. Second, cache line data movements that are on the critical path are
quickly served, and have fewer slowdowns from expensive page movements that
may have been previously triggered, since _DaeMon_ effectively prioritizes
cache line movements.

(II) _Selection Granularity Data Movement_. To handle network, architecture
and application variability in disaggregated systems, we design a dynamic
approach to decide whether a request should be served by a cache line, page,
or _both_ , depending on application and network characteristics. At _DaeMon_
’s engine of each compute component, we include two separate hardware buffers
to track _pending_ data migrations for both cache line and page granularity,
and a selection granularity unit to control the granularity of upcoming data
requests based on the utilization of the above buffers. The utilization of
these buffers allows us to capture dynamic information regarding the current
traffic in the system and the application behavior (i.e., locality). Our
proposed selection granularity data movement enables robustness against
fluctuations in network, architecture and application characteristics (we
explain how this is implemented in §4.2).

(III) _Link Compression on Page Movements_. We leverage the decoupled page
movement to use a high-latency link compression scheme (with high compression
ratio), when moving pages across the network. We integrate hardware
compression units at both the compute components and memory components to
highly compress pages moved over the network: the page is compressed before it
is being transferred over the network, and decompressed when it arrives at the
destination (before it is written in memory modules). _Link compression_ on
page movements reduces the network bandwidth consumption and alleviates
network bottlenecks.

Overall, _DaeMon_ cooperatively integrates all three key techniques, the
synergy of which provides a versatile solution:
(1) Prioritizing requested cache lines helps _DaeMon_ to tolerate high
(de)compression latencies in page migrations over the network, while also
leveraging benefits of page migrations (low metadata overheads, spatial
locality).
(2) Moving compressed pages consumes less network bandwidth, helping _DaeMon_
to reserve part of the bandwidth to effectively prioritize critical path cache
line accesses.
(3) Selection granularity movement helps _DaeMon_ to adapt to the application
data compressibility: if the pages are highly compressible, the number of
_pending_ page migrations is relatively low, thus _DaeMon_ favors moving data
more at page granularity instead of cache line granularity (and vice-versa).

##  4\. _DaeMon_ : Detailed Design

We design _DaeMon_ to be a disaggregated solution: a _DaeMon_ compute engine
is added at _each_ compute component of the system to handle data requests to
remote memory, and a _DaeMon_ memory engine is integrated at the controller of
_each_ memory component of the system. Figure 5 shows our proposed
architecture.

Figure 5. Proposed architecture for compute component and memory component.

The baseline architecture of each compute component includes a chiplet-based
CPU+FPGA architecture (this CPU+FPGA integrated design has also been proposed
to prior state-of-the-art work (Calciu et al., 2021; Hwang et al., 2020)),
which is expected to have small cost (Calciu et al., 2021) compared to the
overall cost savings enabled by disaggregated systems, while it is also socket
compatible to current systems (Hwang et al., 2020; Cock et al., 2022). The
FPGA has three communication paths: i) a coherent path, i.e., CPU-FPGA
coherent links, to access the CPU on-chip cache hierarchy, ii) an interface
(channel-based connection) to access the local memory, and iii) an external
connection to the network controller to move data to/from remote memory. We
propose extending the FPGA by adding a new lightweight hardware component to
handle data requests, i.e., the _DaeMon_ compute engine.

Each memory component includes its own controller (Shan et al., 2018; Lee et
al., 2021; Guo et al., 2022), that has two communication paths: a channel-
based connection to DIMM modules of remote memory, and an external connection
to the network, which is used to move data from/to compute components. We
propose extending the controller of each memory component by adding a new
hardware component to handle data movements, i.e., the _DaeMon_ memory engine.

In our study, we assume that the local memory is an inclusive cache for the
remote memory, which contains all application data. The local memory
implements an approximate LRU replacement policy, similar to prior state-of-
the-art work (Shan et al., 2018).

###  4.1. Enabling Decoupled Multiple Granularity Data Movement

Figure 6 shows the detailed design of the _DaeMon_ compute engine and _DaeMon_
memory engine. _DaeMon_ engine includes two queues to handle requests at each
granularity: cache line granularity via the sub-block queue 1, and page
granularity via the page queue 2. It also includes a queue controller 7 to
serve requests from both queues, and a packet buffer 6 to temporarily keep
arrived packets, while they are being processed.

Figure 6. Detailed design of _DaeMon_ engines for the compute (left) and
memory (right) components.

Approximate Bandwidth Partitioning. To prioritize cache line data movements
while also ensuring that page movements are not aggressively stalled, we
design an approximate bandwidth partitioning approach between the cache line
and page movements, and configure the queue controller to serve cache line and
page requests with a _predefined fixed ratio_. Assuming that cache line and
page requests transfer 64B and 4KB of data, respectively, and having a
bandwidth partitioning ratio of 25% (Figure 11 presents a sensitivity study on
this ratio), 25% of the bandwidth is reserved for cache lines as follows: for
each page request issued through the network, which results in transferring
4KB data, the queue controller needs to serve $4096/64*0.25/(1-0.25)\approx
21$ cache line requests, each transferring 64B of data. To ensure this
approximate partitioning is always maintained, we retain this alternate
serving of page and cache line requests even if either queue is empty (i.e.,
requests may be not issued in all cycles). _DaeMon_ implements an approximate
bandwidth partitioning both in the network across components of the system and
when accessing data from remote memory modules.

###  4.2. Selecting the Data Movement Granularity

_DaeMon_ compute engine additionally includes two separate hardware buffers to
track data requests which are scheduled to be moved or in the process of being
migrated (henceforth referred to as inflight): (i) the inflight sub-block
buffer for the cache line granularity requests 3, and (ii) the inflight page
buffer for the page granularity requests 4. Both buffers are used to track
_pending_ data migrations and avoid requesting the same data multiple times.
_DaeMon_ compute engine includes a selection granularity unit 5 which
throttles data requests to avoid requesting the same data multiple times, and
decides at which granularity the request should be served (cache line, page,
or both granularities).

Scheduling Page Granularity Data Movements. When the _DaeMon_ compute engine
receives a data request, the selection granularity unit checks (i) the
utilization of the inflight page buffer, and (ii) if the corresponding page
has already been scheduled to be moved. If the page has already been requested
or the inflight page buffer is full, the selection granularity unit _does not
request the page_. Thus at any given time, the number of pages scheduled to be
moved is automatically limited by the selection granularity unit, also
limiting storage/area overheads to track the pending page migrations.

If the inflight page buffer is not full, the selection granularity units
schedules the page migration by adding a new entry in the page queue and the
inflight page buffer, marking the page as scheduled. When the queue controller
issues the movement, the corresponding entry is released in the page queue,
and the page entry in the inflight page buffer is marked as moved. When the
requested page arrives, the corresponding entry is released (invalid state) in
the inflight page buffer. The page is written to local memory and all pending
requests are serviced via local memory. Any entries in the inflight sub-block
buffer with requests to cache lines in the same page are removed and thus, any
data packets that arrive in the future with cache lines from the same page are
simply ignored. In _DaeMon_ , we retain existing data management and address
translation mechanisms at page granularity. Local page table updates at the
compute component are only performed in page migrations.

Scheduling Cache Line Granularity Data Movements. To decide whether a cache
line granularity movement should be made, the selection granularity unit
checks (i) the utilization of the inflight sub-block buffer and (ii) if the
corresponding page was already scheduled to be moved (by a previous request).
There are two cases. First, if the corresponding page is _not_ scheduled to be
moved according to the inflight page buffer, the selection granularity unit
always schedules a cache line granularity data movement. Second, if the
corresponding page is already scheduled to be moved, the selection granularity
unit sends the cache line only if: (i) the sub-block buffer has lower
utilization than the page buffer and (ii) the page is not already in the
process of migration (i.e., the page is in the page queue). Otherwise, it
drops the request as the page has already been requested. This avoids
unnecessarily sending cache lines when the corresponding page is likely to
arrive faster and when the sub-block queue is likely to be slow due to
oversaturation.

If a cache line is scheduled, a new entry is added both in the sub-block queue
and the inflight sub-block buffer. When the queue controller issues the
movement, the corresponding entry is released in the sub-block queue. When the
requested cache line arrives at the compute component, the corresponding entry
is released in the sub-block buffer, and the data is _directly_ written to LLC
through the FPGA-based coherent interconnect.

The above mechanism enables an adaptive approach for the data movement
granularity based on the dynamic network/architecture and application
characteristics:

(1) If there is _high locality_ within pages, there are fewer pages requested,
and the sub-block buffer fills up faster than the page buffer. Thus, _DaeMon_
favors issuing pages and throttles cache line requests. If there is _low
locality_ within pages, the page buffer fills up faster than the sub-block
buffer, since cache line requests are served at a higher rate than page
requests (e.g., 21:1 cache lines versus pages requests for 25% bandwidth
ratio). Thus, _DaeMon_ favors issuing cache line movements and throttles page
migrations.

(2) If both the page and sub-block buffers are fully utilized, _DaeMon_
detects _bandwidth constrained_ scenarios. In bandwidth constrained scenarios,
_DaeMon_ favors issuing more cache line movements to alleviate bandwidth
bottlenecks. When the bottleneck is mitigated, (inflight buffers are not fully
utilized), _DaeMon_ schedules more page movements to obtain locality benefits.

(3) Additionally, when using link compression to transfer pages, _DaeMon_ is
able to adapt to the compressibility of the application data: if the pages are
highly compressible, the inflight page buffer empties at a faster rate and
thus _DaeMon_ favors sending more page migrations (and vice versa).

###  4.3. Handling Dirty Data

Dirty data (cache lines/pages) is always directly written to remote memory.
Data (cache line or page granularity) can be in one of the three states: (i)
local: when data is cached in on-chip caches (for cache lines) or local memory
(for pages), (ii) remote: when data is only in remote memory, and (iii)
inflight: when data is being migrated. With _DaeMon_ , data can be present
simultaneously in two states: for example, local as a cache line (in the cache
hierarchy of the compute component) and inflight as a page or vice versa. This
poses coherence issues if the processor writes to data in the above state.

There are two scenarios: (i) if a page arrives to compute component _before_ a
prioritized cache line, any modifications to the page may be overwritten by
the stale cache line that arrives later, and (ii) if a dirty cache line is
evicted from the LLC while the corresponding page is in transit, the
modifications would be lost when the page arrives to compute component. As
explained, in the (i) scenario, when a page arrives, the corresponding entries
in the inflight sub-block buffer with requests to cache lines in the same page
are removed and thus, any data packets that arrive in the future with cache
lines from the same page are simply ignored. In the (ii) scenario, for every
dirty cache line that gets evicted by the LLC and also misses in the local
memory, its corresponding page can be either inflight or in remote memory. To
ensure correctness, _DaeMon_ compute engine first checks if there is an
inflight page request in the inflight page buffer. If there is _no_ inflight
page request (according to the inflight page buffer), the evicted dirty cache
line is directly migrated to remote memory. In the other case, we need to
retain the dirty cache line until the page arrives. We include a dirty unit 8
in the _DaeMon_ compute engine with a _dirty data buffer_ that temporarily
stores these dirty cache lines. When the corresponding inflight page arrives,
the _DaeMon_ compute engine flushes the dirty cache line(s) from the dirty
buffer to local memory.

Prior works (Calciu et al., 2021; Adya et al., 2019) observe that typically a
few cache lines (1-8 cache lines) or all cache lines of a page are accessed.
Thus, when the evicted dirty cache lines of the same page increase beyond a
predefined threshold (e.g., 8 cache lines), the _DaeMon_ compute engine
flushes all dirty cache lines to remote memory, and marks the corresponding
entry for that page in the inflight page buffer as _throttled_. When the
inflight page arrives, the _DaeMon_ compute engine ignores it, since its entry
is in the _throttled_ state, and sends a new request for that page to receive
the up-to-date data. This enables lower area/storage overheads for the dirty
data buffer.

###  4.4. Link Compression in Page Migrations

Approaches for data compression are typically of two types: (i) latency-
optimized compression schemes (Pekhimenko et al., 2012; Alameldeen and Wood,
2004b; Chen et al., 2010; Yang et al., 2000, 2004), which optimize/minimize
the (de)compression latencies, and (ii) ratio-optimized compression schemes
(Abali et al., 2001; Tremaine et al., 2001; Kim et al., 2017; Ziv and Lempel,
1977), which provide higher compression ratios while incurring relatively high
(de)compression latencies. We select a ratio-optimized compression scheme in
_DaeMon_ based on two observations (§6): (i) in disaggregated systems,
queueing delays and network latencies can be significant, thus compression
benefits outweigh the high (de)compression latencies, and (ii) _DaeMon_
prioritizes cache lines that are on the critical path, thus we can tolerate
relatively high (de)compression latencies for page migrations.

_DaeMon_ engines include (de)compression units 9 10 that compress pages
transferred through the network. We implement a hardware design similar to IBM
MXT (Abali et al., 2001; Tremaine et al., 2001), using the LZ77 compression
algorithm (Ziv and Lempel, 1977), and operating at 1KB granularity at a time.
(De)Compression units include 4 engines, each of which operates on 256B of
data and uses a 256B shared dictionary, incurring in total a 64-cycle latency
according to (Abali et al., 2001; Tremaine et al., 2001).

###  4.5. _DaeMon_ ’s Hardware Structures

We estimate the overheads of _DaeMon_ ’s hardware structures for each compute
component assuming a 64-core CPU, using CACTI (Muralimanohar et al., 2007).
The sizes of the _DaeMon_ sub-block and page queues and the sub-block and page
buffers have been selected based on the maximum possible number of _pending_
data migrations at a time, which is determined by the number of the available
LLC MSHRs (Miss Status Holding Registers) in a typical CPU system, and is
independent of the workloads’ patterns and the mix of workloads that are
running at each time. For the hardware structures at each memory component, we
scale the sizes of the _DaeMon_ sub-block and page queues, assuming that each
memory component can concurrently serve up to 4 compute components. Table 1
presents the hardware overheads of _DaeMon_ compute engine (C) and _DaeMon_
memory engine (M). Figure 7 shows an inflight sub-block buffer entry, an
inflight page buffer entry, and a dirty data buffer entry.

Hardware | Entries | Size | Access | Area | Energy
---|---|---|---|---|---
Structure |  | (KB) | Cost (ns) | Cost (mm2) | Cost (nJ)
Sub-block Queue (C) | 128 | 0.5 | 0.34 | 0.084 | 0.038
Sub-block Queue (M) | 512 | 2 | 0.38 | 0.093 | 0.039
Page Queue (C) | 256 | 1 | 0.35 | 0.087 | 0.038
Page Queue (M) | 1024 | 4 | 0.40 | 0.105 | 0.041
Inflight Sub-block Buffer (C) | 128 | 1.625 | 0.56 | 0.041 | 0.046
Inflight Page Buffer (C) | 256 | 3.25 | 0.77 | 0.089 | 0.096
Dirty Data Buffer (C) | 256 | 17 | 0.62 | 0.168 | 0.046
Packet Buffer (C) | - | 8 | 0.538 | 0.137 | 0.044
Packet Buffer (M) | - | 32 | 1.032 | 0.263 | 0.047
2 $\times$ Dictionary Table (C,M) | 1024 | 1 | 0.28 | 0.015 | 0.020

Table 1. _DaeMon_ ’s hardware overheads for C: compute engine and M: memory
engine. Figure 7. An inflight sub-block buffer entry, an inflight page buffer
entry, and a dirty data buffer entry.

Sub-block Queue (SRAM), 128 entries: The sub-block queue size is limited by
the available LLC MSHRs of the compute component.
Page Queue (SRAM) - 256 entries: The page queue has 256 entries, since
_DaeMon_ serves requests from the page queue at a smaller rate than the sub-
block queue.
Inflight Sub-block Buffer (CAM) - 128 entries: Similar to the sub-block queue,
this buffer has 128 entries. We design this hardware structure to be indexed
using the corresponding page address to achieve smaller area costs, since at a
given time there may be multiple inflight cache line requests to the same
page. Each entry (Figure 7a) includes the page address, the state (_scheduled_
or _invalid_), and a 64-bit queue that is used to indicate the offsets within
the page of the inflight cache requests by (re)setting the corresponding bits.
Inflight Page Buffer (CAM) - 256 entries: An inflight page buffer entry
(Figure 7b) includes the page address, the state that can be _scheduled_ ,
_moved_ , _throttled_ (when the page needs to be re-requested) or _invalid_ ,
and a 64-bit queue to indicate the offsets of the dirty cache lines of the
inflight page that are temporarily kept in the dirty data buffer.
Dirty Data Buffer (SRAM) - 256 entries: A dirty data buffer entry (Figure 7c)
includes the evicted cache line and its address.
Packet Buffer (SRAM) - 8KB: We use an 8KB buffer to temporarily store arrived
data packets until they are processed.
Dictionary Tables for (De)Compression (CAM) - 2KB: _DaeMon_ proposes 4 engines
at each (de)compression unit, each of them has 256B CAM (Abali et al., 2001;
Tremaine et al., 2001). In total, we estimate each dictionary table as 1KB
CAM.

Overall, _DaeMon_ ’s hardware overheads are due to the cache memories
corresponding to the sub-block and page queues, the sub-block and page
buffers, and the dictionary tables used for data compression. The total sizes
of the _DaeMon_ cache memories are $\sim$34KB and 40KB for the _DaeMon_
compute and memory engine, respectively. Therefore, _DaeMon_ ’s hardware
overheads are similar to that of the small L1 cache memory of a modern state-
of-the-art processor (e.g., Intel Xeon). We conclude that our proposed
hardware structures incur very modest hardware and financial costs to be
integrated into the compute components and memory components of disaggregated
systems.

###  4.6. Handling Failures

_DaeMon_ handles compute component, memory component and network failures
using fault-tolerance approaches of prior works (Lee et al., 2021; Shan et
al., 2018; Calciu et al., 2021). If the compute component fails (CPU or
_DaeMon_ compute engine), the application needs to be restarted potentially on
a different compute component of the system. Network failures are handled
using timeouts: _DaeMon_ engines can trigger timeouts when pending page or
cache line requests have not arrived after a long time, or when ACK messages
have not been received for migrations of dirty data. The exploration of the
timeout period value is left for future work. Finally, memory component
failures are handled via data replication, similarly to prior work (Shan et
al., 2018): _DaeMon_ can send the evicted dirty data to more than one memory
component, and wait to receive ACK messages from all of them.

###  4.7. _DaeMon_ Extensions

Prefetching. _DaeMon_ can flexibly support hardware/software-based
prefetchers. Existing CPU prefetchers might generate data requests, which
_DaeMon_ can normally serve by migrating the prefetched data at a cache line
granularity, page granularity or both granularities, via on our proposed
selection granularity scheme. Page prefetchers (Maruf and Chowdhury, 2020)
might generate page-granularity data requests, which _DaeMon_ can serve by
migrating the prefetched data at page granularity or throttling the page
request based on our selection granularity scheme.
Large Pages. _DaeMon_ can be easily extended to support large granularity
pages (e.g., 2MB). To effectively prioritize cache line requests over page
requests, _DaeMon_ ’s predefined ratio for the approximate bandwidth
partitioning needs to be properly configured based on the size of the large
page. To enable multiple page sizes (e.g., both 4KB and 2MB), we could enhance
_DaeMon_ to split large pages (e.g., 2MB) to consecutive page requests of
smaller sizes (e.g., 4KB) issued in the page queue.

##  5\. Methodology

Simulation Methodology. We use Sniper (Carlson et al., 2011; Carlson et al.,
2014), a state-of-the-art accurate simulator, and we heavily modified it to
model a disaggregated system with one compute component and multiple memory
components interconnected across the network. We present detailed evaluation
results using one memory component and provide a characterization study of
multiple memory components with various network configurations in Figure 17.
For the network across components, we use both (i) a fixed latency of 100
ns/400 ns (Gao et al., 2016; Lee et al., 2021) to model propagation and
switching delays inside network (referred to as _switch latency_), and (ii) a
variable latency of modeling the current bandwidth utilization at each
simulation interval (100K ns) when configuring the network bandwidth to be
2-8$\times$ less than DRAM bandwidth (Shan et al., 2018; Gao et al., 2016)
(referred to as _bandwidth factor_). For the compute component, we configure a
state-of-the-art CPU server with on-chip cache memories of typical sizes and
x86 OoO cores of 3.6GHz frequency. The local memory size is configured to fit
$\sim$20% of each application’s working set, and we evaluate LRU replacement
policy (Shan et al., 2018) in local memory, unless otherwise stated. The
aforementioned configuration is consistent with prior state-of-the-art works
in disaggregated systems (Shan et al., 2018; Lee et al., 2021; Gao et al.,
2016). For both the local memory and remote memory, we evaluate a DDR4 memory
model with 17GB/s bus bandwidth, and we simulate hardware-based address
translation for memory pages, having overhead as one DRAM access cost per
lookup, as explained in prior state-of-the-art work (Guo et al., 2022). We
evaluate access overheads in _DaeMon_ queues/buffers using CACTI
(Muralimanohar et al., 2007) (See Table 1). Table 2 lists the parameters of
our simulated system.

CPU | 3.6 GHz, 4-way OoO x86 cores, 224-entry ROB;
---|---
L1 Instr. Cache | 32 KB, 4-way associativity, LRU;
L1 Data Cache | 32 KB, 8-way associativity, 4-cycle access latency, LRU;
L2 Cache | 256 KB, 8-way associativity, 8-cycle access latency, LRU;
LLC | 4MB, 16-way associativity, 30-cycle access latency, LRU;
Local Memory | 2400MHz, 15ns process. latency, 17GB/s bus bandwidth (Assn., 2017);
Network | 2-8$\times$ less than bus bandwidth, 100-400 ns switching latency (Gao et al., 2016; Shan et al., 2018);
Remote Memory | 2400MHz, 15ns process. latency, 17GB/s bus bandwidth (Assn., 2017);

Table 2. Configuration of simulated system.

Workloads. We evaluate various workloads with different memory access patterns
from various application domains including graph processing, machine learning,
bioinformatics, linear algebra, data analytics, and HPC domains, shown in
Table 3. The dynamic working sets at any given point at runtime range from
43.2MB to 1.32GB. In a fully disaggregated system, the application working set
(irrespective of the size) is primarily housed in remote memory to provide the
benefits of improved elasticity, heterogeneity, and failure isolation.
Therefore, we configure the local memory size to fit $\sim$20% of each
application’s working set (similar to prior state-of-the-art work (Shan et
al., 2018; Lee et al., 2021; Gao et al., 2016)). All data is initially located
in remote memory. We simulate most workloads to full execution and for slower
long running workloads, we simulate 1B instructions.

Workload | Domain | Input Data
---|---|---
K-Core Decomposition (kc) (Shun and Blelloch, 2013) | Graph Processing | 1M vertices x 10M edges
Triangle Counting (tr) (Shun and Blelloch, 2013) | Graph Processing | 1M vertices x 10M edges
Page Rank (pr) (Shun and Blelloch, 2013) | Graph Processing | 1M vertices x 10M edges
Needle Wunsch (nw) (Che et al., 2009) | Bioinformatics | 4096 base pairs per sequence
Breath First Search (bf) (Shun and Blelloch, 2013) | Graph Processing | 1M vertices x 10M edges
Betweenness Centrality (bc) (Shun and Blelloch, 2013) | Graph Processing | 1M vertices x 10M edges
Timeseries (ts) (Yeh et al., 2016) | Data Analytics | 262144 elements in sequence
Sparse Matrix Vector Multipl. (sp) (Kjolstad et al., 2017) | Linear Algebra | pkustk14 matrix
Sparse Lengths Sum (sl) (Naumov et al., 2019) | Machine Learning | Kaggle Criteo 10GB Dataset
High Perf. Conjugate Gradient (hp) (HPCG, 2019) | HPC | 104 x 104 x 104
Particle Filter (pf) (Che et al., 2009) | HPC | 4096 x 4096, 30000 particles
Darknet19 (dr) (Redmon, 2016) | Machine Learning | dog.jpg (768 x 576 pixels)
Resnet50 (rs) (Redmon, 2016) | Machine Learning | dog.jpg (768 x 576 pixels)

Table 3.  Summary of workloads.

##  6\. Evaluation

We evaluate six schemes: (i) Remote: the typically-used approach (Shan et al.,
2018; Lee et al., 2021; Calciu et al., 2021) of moving data to/from remote
memory at page granularity; (ii) LC: _DaeMon_ ’s link compression for page
movement without enabling cache line granularity data movement, i.e., moving
data at page granularity with LZ-based link compression enabled; (iii) BP:
enabling only _DaeMon_ ’s decoupled multiple granularity data movement with
25% bandwidth partitioning ratio for cache line movements, i.e., moving data
_always_ at both granularities; (iv) PQ: enabling _DaeMon_ ’s _both_ decoupled
multiple granularity and selection granularity data movement with 25%
bandwidth partitioning ratio for cache line movements (without enabling data
compression in page migrations); (v) _DaeMon_ : _DaeMon_ ’s complete design
enabling all its three techniques (25% bandwidth partitioning ratio); and (vi)
Local: the monolithic approach where all the data fits in local memory of the
compute component.

###  6.1. Performance

Figure 8 compares all schemes with different network configurations. Our
evaluated workloads exhibit three patterns and we make the following
observations.

Figure 8. Speedup in all workloads normalized to Remote using various network
configurations.

First, _kc, tr, pr, and nw_ exhibit relatively poor spatial locality within
pages. In such workloads, BP effectively prioritizes critical cache line
requests. However, PQ provides significant benefits thanks to dynamically
selecting the data movement granularity: the page buffer saturates faster than
the sub-block buffer given the poor locality and the higher servicing rate of
the cache line requests in the queue controller, thus the selection
granularity unit enables the movement of more cache lines and fewer pages.
This results in reduced access latencies as critical path cache line requests
are no longer stalled behind many page migrations.

Second, _bf, bc, and ts_ exhibit medium spatial locality within pages. In such
workloads, both LC and PQ decrease data access costs using different
approaches: LC enables exploiting more spatial locality by moving more pages,
while PQ accelerates accesses to the critical path cache line requests, both
of which benefit these workloads.

Third, the remaining workloads exhibit high spatial locality within pages,
thus page migration is critical to leverage data locality. In these workloads,
BP incurs high performance slowdowns, since it is _oblivious_ to application
behavior. Instead, PQ effectively enables more page movements and throttles
cache line movements by tracking pending data requests, thus achieving similar
system performance to Remote. LC performs better for _sp, sl, hp, and pf_ ,
since these workloads have higher data compressibility than _dr_ and _rs_.

Fourth, when network bandwidth is more constrained, LC provides _even higher_
performance over Remote, while PQ is unaffected by bandwidth as the bandwidth
partitioning approach prioritizes cache line movements even with low available
bandwidth.

Fifth, PQ is slightly affected by the switch latency (Please also see Figure
20 in Appendix § A): PQ outperforms Remote by 1.60$\times$ and 1.51$\times$
for 100 ns and 400 ns switch latency, respectively. The slightly lower
benefits are due to PQ’s inability to hide network switch latencies in
critical cache line movements. Instead, LC is unaffected by switch latency, as
page movement incurs much higher overheads (due to very high network
processing and queueing delays) over the smaller switch latency, which link
compression is able to alleviate.

Finally, _DaeMon_ provides high performance benefits for all three classes of
workloads with different locality characteristics thanks to synergistically
integrating both LC and PQ: (i) PQ helps hide the (de)compression latencies in
LC and migrate fewer pages in order to prioritize critical path cache line
movements, and (ii) LC releases network bandwidth resources and helps recover
the lost spatial locality in pages by moving more pages with the available
network bandwidth. _dr_ and _rs_ show only 1.05$\times$ speedup over Remote as
neither LC nor PQ is able to provide speedups due to the poor application data
compressibility and high spatial locality within pages (which favors moving
pages rather than cache lines). _DaeMon_ ’s adaptive approach also provides
high performance benefits across all network configurations: (i) when the
switch latencies are high, cache lines movements are slowed down and the sub-
block queue fills up faster, thus _DaeMon_ favors moving more pages, which is
more effective at high network switch latencies; and (ii) the approximate
bandwidth partitioning approach effectively prioritizes cache line over page
movements even when network bandwidth is constrained. Therefore, _DaeMon_
significantly outperforms the state-of-the-art Remote scheme by 1.85$\times$,
2.36$\times$, 2.97$\times$ for 1/2, 1/4, and 1/8 bandwidth factor,
respectively.

Overall, we conclude that _DaeMon_ ’s cooperative techniques provide a robust
approach to alleviate data movement overheads across various network
characteristics and application behavior.

###  6.2. Memory Access Costs

Figure 9 compares the average data access costs (latencies) achieved by
various schemes normalized to Remote. Due to space limitations, in the
remaining plots, we present a _representative_ subset of our evaluated
workloads, but we report geometric mean values across _all_ evaluated
workloads. Please also see Figure 19 in Appendix § A, which compares the
network bandwidth utilization achieved by the various data movement schemes.

Figure 9. Data access costs achieved by various schemes normalized to Remote.

We make three observations. First, LC improves data access costs over Remote
by 2.12$\times$ across all network configurations (not graphed), because it
reduces the network processing costs and queueing delays by sending fewer
bytes through the network. PQ improves access costs (2.06$\times$ over Remote
across all configurations) by prioritizing critical path cache line movements.
Second, PQ _significantly_ reduces data access costs in workloads with poor
page locality (e.g., pr, nw), since critical path cache line movements are not
stalled by migrating pages. However, in applications with high data locality
(e.g., dr, rs), although PQ reduces data access costs by 1.43$\times$ over
Remote, it improves performance by only 1.05$\times$, because the selection
granularity unit favors sending pages for workloads with high locality and a
few requests are served at cache line granularity. Third, _DaeMon_
significantly reduces data access costs by 3.06$\times$ over Remote. _DaeMon_
employs link compression to migrate more pages with lower network overhead
over PQ, thus exploiting more data locality, while also leveraging the ability
to prioritize critical cache line requests. In _pr_ , _DaeMon_ can achieve
lower access latency than Local, since serving requests from both local memory
and remote memory increases the effective aggregate memory bandwidth.

###  6.3. Hit Ratio in Local Memory

Figure 10 presents the hit ratio in local memory, and is thus a measure of the
page movement benefits. To prioritize cache lines, PQ throttles some page
migrations, thus reducing the local memory hit rate as a tradeoff for reduced
access latencies to critical path cache line requests. However, _DaeMon_
enables moving more pages over PQ thanks to link compression, while still
retaining the cache line prioritization benefits of PQ. The numbers shown over
each bar for _DaeMon_ present the additional pages that were moved in _DaeMon_
as a percentage over PQ, thanks to the reduced bandwidth consumption provided
by link compression. A zero value indicates that neither PQ nor _DaeMon_ has
throttled any page movement.

Figure 10. Hit ratio in local memory achieved by various schemes.

We draw three findings. First, Remote has on average 97.7% hit ratio in local
memory. Thanks to high spatial locality, all workloads benefit from page
migration, leading to high hit rates: even workloads with relatively poor
spatial locality (e.g., _nw_) have 90% hit ratio in local memory. Second, PQ
decreases the hit ratio in local memory by up to 18.4% over Remote, because PQ
throttles page movements in some workloads to prioritize cache line requests,
thus increasing the number of accesses to remote memory. Third, _DaeMon_
recovers most of the lost local memory hits, achieving on average _only_ 0.4%
worse hit ratio over Remote. Leveraging link compression in _DaeMon_ reduces
network bandwidth consumption and significantly increases the number of pages
that can be migrated over PQ. Across all configurations (not graphed) _DaeMon_
migrates 68.9% of the pages throttled by PQ via leveraging link compression.
We conclude that _DaeMon_ enables both leveraging the benefits of data
locality within pages and the prioritization of critical path cache line
requests.

###  6.4. Sensitivity Study to Bandwidth Partitioning Ratio

Figure 11 presents a sensitivity study on the bandwidth partitioning ratio
between the cache line and page movements.

Figure 11. Performance of PQ and _DaeMon_ normalized to Remote varying the
bandwidth partitioning ratio.

We draw three findings. First, a higher bandwidth partitioning ratio (e.g.,
50%) than _DaeMon_ ’s default 25% ratio, incurs slowdowns in workloads of
medium and high spatial locality, and only improves performance in workloads
with very low locality within pages (e.g., _pr, nw_). This is because high
bandwidth partitioning ratios favor cache line movements and throttle a higher
number of page movements. Second, since cache line data movements are affected
more by the switch latency compared to page movements, the performance
benefits of higher bandwidth partitioning ratios reduce at higher switch
latencies. For example, in _pr_ , the 50% bandwidth partitioning ratio
outperforms 25% ratio by 1.19$\times$ and 1.08$\times$ using _DaeMon_ at 100
ns and 400 ns switch latency, respectively. Finally, across all different
bandwidth factors (not graphed), _DaeMon_ ’s default 25% ratio outperforms the
50% ratio by 1.02$\times$ and 1.04$\times$ for 100 ns and 400 ns switch
latency, respectively, and the 80% ratio by 1.07$\times$ and 1.33$\times$ for
100 ns and 400 ns switch latency, respectively. We conclude that _DaeMon_ ’s
default 25% ratio on average performs best across all various network and
application characteristics.

###  6.5. Sensitivity Study to Various Compression Algorithms

Figure 12 compares the performance of LC normalized to Remote with three
compression schemes: (i) fpcbdi: a latency-optimized hybrid scheme of BDI
(Pekhimenko et al., 2012) and FPC (Alameldeen and Wood, 2004b) with 4-cycle
(de)compression latency per cache line (Kim et al., 2017); (ii) fve: the
latency-optimized FVE (Thuresson and Stenström, 2008) scheme using a 256B
dictionary table and having 6-cycle (de)compression latency per cache line
(Thuresson and Stenström, 2008); and (iii) LZ: _DaeMon_ ’s compression ratio-
optimized LZ-based scheme (Kim et al., 2017; Tremaine et al., 2001) (See
details on § 4.4).

Figure 12. Performance of LC varying the compression scheme.

We observe that LZ always outperforms Remote, despite the high (de)compression
latencies, because the network overheads are significantly higher, indicating
that link compression is a highly effective solution for disaggregated
systems. dr and rs show little performance improvement with LZ, because the
application data is less compressible (their compression ratio is 1.42$\times$
versus 4.47$\times$ on average across all evaluated workloads). Moreover, LZ
outperforms fpcbdi and fve across all network configurations (not graphed) by
1.54$\times$ and 1.44$\times$ on average, respectively, since it achieves
higher compression ratios (on average 2.92$\times$ and 2.73$\times$ higher
compression ratio than fpcbdi and fve respectively). The benefits of LZ over
fpcbdi and fve are even higher in the more bandwidth limited configurations
(e.g., with 1/8 bandwidth factor). Therefore, we conclude that the high
network overheads in disaggregated systems favor compression algorithms that
provide higher compression ratios, since the benefits of the reduced bandwidth
consumption outweigh the higher (de)compression latencies.

###  6.6. Network Disturbance Study

Figures 13 and 14 compare the IPC and the hit ratio in local memory
respectively,  of LC, PQ and _DaeMon_ , when the network traffic varies during
runtime: we simulate contention from other compute components that share the
same network, by artificially injecting packets inside the network. We
evaluate _pr_ and _nw_ , as they incur the highest data movement costs.

Figure 13. Performance of LC, PQ, _DaeMon_ , when creating artificial
disturbance in the network during runtime.

Figure 14. Hit ratio in local memory of LC, PQ and _DaeMon_ , when creating
artificial disturbance in the network during runtime.

_DaeMon_ outperforms both LC and PQ by 2.85$\times$ and 1.19$\times$,
respectively, even when network traffic varies during runtime. _DaeMon_
effectively adapts to varying application behavior and network conditions at
runtime. For example, in _nw_ , in the first 50M instructions, _DaeMon_
benefits more from LC as the application has high bandwidth consumption and
higher locality within pages. In the next 100M instructions, the workload
exhibits less data locality within pages, and _DaeMon_ benefits more from PQ,
which provides significant performance benefits over LC by effectively
prioritizing critical path cache line requests. In the last part of execution,
_DaeMon_ again leverages the benefits of LC. Therefore, we conclude that
_DaeMon_ provides a versatile approach to dynamic and variable runtime
application and network characteristics.

###  6.7. Multithreaded Performance

Figure 15 shows _DaeMon_ ’s performance benefits for multithreaded workloads
on 8 OoO cores, thus evaluating more bandwidth-limited executions compared to
that of Figure 8. Please also see Figure 21 in Appendix § A which evaluates
even more bandwidth-limited executions. Across _all_ workloads and network
configurations (not graphed), _DaeMon_ outperforms the typically-used Remote
scheme by 2.73$\times$ on average. When network bandwidth is very limited,
e.g., 1/16 bandwidth factor (Figure 21), _DaeMon_ ’s benefits are even higher,
by 3.95$\times$ over Remote.

Figure 15. Speedup achieved by various schemes in multithreaded workloads
normalized to Remote.

###  6.8. Sensitivity Study to Replacement Policy in Local Memory

Figure 16 compares _DaeMon_ and Local normalized to Remote, when using First-
In-First-Out (FIFO) replacement policy in local memory.

Figure 16. Performance of Local and _DaeMon_ over Remote, when using FIFO
replacement policy in local memory.

Across _all_ workloads and network configurations (not graphed), _DaeMon_
outperforms the widely-adopted Remote scheme by 2.63$\times$, when using a
FIFO replacement policy in local memory. _DaeMon_ is orthogonal to the
replacement policy used in local memory, and can be used synergistically with
any arbitrary replacement policy in local memory to even further reduce data
access costs. Overall, _DaeMon_ can significantly mitigate the data movement
overheads in fully disaggregated systems independently on the number of data
migrations happens during runtime: even when a small number of data migrations
happens during runtime (e.g., thanks to sophisticated approaches such as
intelligent replacement policies in local memory, hot page placement/selection
techniques, page prefetchers), _DaeMon_ can even further alleviate the data
movement costs by dynamically selecting the granularity of data movements,
prioritizing the critical cache line requests, and opportunistically moving
compressed pages at slower rates. Therefore, we conclude that _DaeMon_ can
work synergistically with sophisticated replacement policies in local memory,
page prefetchers and intelligent page placement/movement techniques to even
further improve system performance.

###  6.9. Sensitivity Study to Multiple Memory Components

Figure 17 compares Remote and _DaeMon_ normalized to Local, when varying the
number of memory components and having a different network configuration for
each memory component. Please also see Figure 22 in Appendix § A. We evaluated
distributing memory pages with either a round-robin way or randomly across
remote memory components, and draw the same key observation for both
distributions. When adding more memory components using the same network
configuration with that of when having one memory component (e.g., having 100
ns switch latency and 1/4 bandwidth factor for each memory component),
performance of both Remote and _DaeMon_ improves over Local: memory pages are
distributed across multiple memory components and the system provides larger
aggregate network and memory bandwidth, thus data migrations incur smaller
overheads. Finally, _DaeMon_ significantly outperforms Remote by 3.25$\times$
across all workload-architecture combinations, and constitutes a scalable
solution for large-scale disaggregated systems with multiple hardware
components and various architectures.

| #memory components | stch-lat | bw-fact
---|---|---|---
MC1.1 | 1 | 100 | 1/4
MC2.1 | 2 | 100-100 | 1/4-1/4
MC2.2 | 2 | 400-400 | 1/4-1/8
MC2.3 | 2 | 100-100 | 1/8-1/8
MC4.1 | 4 | 100-100-100-100 | 1/4-1/4-1/4-1/4
MC4.2 | 4 | 100-400-100-400 | 1/4-1/8-1/4-1/8
MC4.3 | 4 | 400-400-400-400 | 1/8-1/8-1/8-1/8
MC4.4 | 4 | 100-100-100-100 | 1/8-1/16-1/8-1/16

Figure 17. Performance of Remote and _DaeMon_ over Local when using multiple
memory components.

###  6.10. Sensitivity Study to Multiple Concurrent Workloads

Figure 18 shows _DaeMon_ ’s performance benefits when concurrently running
multiple workloads on a compute component with 4 OoO cores. The performance of
_each_ core is normalized to that of the same core using Remote. The local
memory hosts $\sim$15% and $\sim$9% of _each_ application’s working set, when
running 2 and 4 workloads, respectively. _DaeMon_ outperforms Remote by
1.96$\times$ across all multiple-workload experiments, thus being highly
efficient and performant when multiple _heterogeneous_ jobs concurrently run
in the disaggregated system.

Figure 18. Performance of _DaeMon_ over Remote when running multiple
concurrent workloads in a 4-CPU compute component and a memory component.

###  6.11. Key Takeaways and Recommendations

This section summarizes our key takeaways and recommendations extracted from
our evaluations.

Key Takeaway #1. There is no one-size-fits-all granularity in data movements:
the best-performing granularity at each time depends on the network/system
load and the application data access patterns, which can significantly vary
across applications and within application during runtime. Figure 8
demonstrates that some applications significantly benefit from the
prioritization of critical cache line data movements (e.g., _pr, nw_), and
some applications only benefit from page migrations that leverage data
locality (e.g., _dr, rs_). Figure 12 shows that some applications have highly
compressible data, and thus greatly benefit from _compressed_ page granularity
data movements. Finally, Figure 13 proves that the application behavior and
network traffic can highly vary during runtime, and thus the best-performing
data movement granularity needs to adapt to the application characteristics
and network/system conditions. Therefore, we recommend that system and
hardware designers of disaggregated systems implement system-level solutions
and hardware mechanisms that dynamically change and adapt their configurations
and selection methods to the availability of the system resources and the
runtime behavior of the heterogeneous applications.

Key Takeaway #2. Typical datacenter applications exhibit high data locality
within memory pages (e.g., 4KB). Figure 10 shows that Remote achieves high
data locality, i.e., always has at least 90% hit ratio in local memory, across
a wide variety of datacenter workloads with diverse access patterns.
Therefore, migrating data at a large granularity, e.g., page granularity, is
very effective and critical to achieving high system performance in fully
disaggregated systems. To this end, we suggest that hardware and system
designers of disaggregated systems retain coarse-grained data migration (i.e.,
page granularity data migration), since it both enables high performance and
maintains low metadata overheads for address translation in local memory and
remote memory.

Key Takeaway #3. Aggressively prioritizing the cache line granularity data
movements that are on the critical path might hurt performance. Figure 11
shows that a high bandwidth partitioning ratio, e.g., 50% or 80% bandwidth
partitioning ratio, which significantly prioritizes the cache line granularity
data movements over the page granularity data movements, incurs significant
performance slowdowns in workloads with medium and high spatial locality. As a
result, we suggest that hardware and system designers of data movement
solutions tailored for disaggregated systems always ensure that page
migrations are not aggressively stalled.

Key Takeaway #4. Distributed and disaggregated data movements solutions are
highly effective and efficient in fully disaggregated systems. disaggregated
systems are distributed architectures and comprise multiple hardware devices,
each of them is independently and transparently managed from other hardware
components in the system. Our evaluations in Figures 17 and 22 show that
distributed and disaggregated solutions for data movement (i.e., _DaeMon_)
better leverage the available aggregate network and memory bandwidth in the
system, and enable high scalability to large-scale disaggregated systems with
multiple hardware components. To this end, we recommend that hardware
architects design distributed hardware mechanisms for fully disaggregated
systems.

##  7\. Related Work

To our knowledge, this is first work to (i) analyze and alleviate the data
movement problem in fully disaggregated systems; (ii) enable prioritized and
decoupled movement of data at multiple granularities simultaneously to reduce
access latencies; (iii) propose a dynamic selection granularity mechanism with
approximate bandwidth partitioning to effectively leverage both cache line and
page movement depending on application and network characteristics; and (iv)
implement a synergistic solution of link compression, bandwidth partitioning,
and adaptive granularity selection in data movements. We discuss prior work.

Disaggregated Systems. Several prior works (Shan et al., 2018; Angel et al.,
2020; Nitu et al., 2018; Han et al., 2013; Calciu et al., 2021; Aguilera et
al., 2017; Zhang et al., [n. d.]; Lim et al., 2012; Wang et al., 2020; Zuo et
al., 2021; Bindschaedler et al., 2020; Peng et al., 2020; Gu et al., 2017;
Aguilera et al., 2018; Lee et al., 2021; Pinto et al., 2020; Katrinis et al.,
2016; Guo et al., 2022; Buragohain et al., 2017; Gao et al., 2016; Rao and
Porter, 2016; Zervas et al., 2018; Gouk et al., 2022; Zhou et al., 2022)
propose OS modules, system-level solutions, programming frameworks, software
management systems, architectures and emulators for disaggregated systems.
These works do not tackle the data movement problem in disaggregated systems,
and thus _DaeMon_ is orthogonal to these proposals. MIND (Lee et al., 2021)
proposes memory sharing among compute components by implementing coherence and
address translation in network switches. Kona (Calciu et al., 2021) is a
software runtime to track cache line granularity accesses to remote memory,
and eliminate page faults by decoupling the application memory access tracking
from the virtual memory page size. However, Kona and MIND do not mitigate data
movement overheads in disaggregated systems, as data is always moved at page
granularity. Thus, _DaeMon_ is largely orthogonal to these works and could be
used to further improve performance. Clio (Guo et al., 2022) proposes a
disaggregated system that virtualizes and manages remote memory at the
hardware level (independently to compute components), and eliminates expensive
page faults in memory components. Clio accesses _remote_ data at a byte
granularity via dedicated API, however _not_ being transparent to programmers.
As explained in § 2.2, moving data _always_ at a small granularity can cause
significant performance penalties in many applications, and does not provide
robustness against fluctuations in network characteristics. Instead, _DaeMon_
is software-transparent, robust and significantly alleviates data movement
costs via decoupled and selective data movement at _multiple_ granularities.
Lim et al. (Lim et al., 2009) propose a disaggregated architecture and
characterize moving data _only_ at cache line or page granularity. The authors
show that the page-based configuration outperforms the cache line
configuration at most common patterns (as observed in § 2.2), however it does
not address the high performance penalties of page migrations. Maruf and
Chowdhury (Maruf and Chowdhury, 2020) propose a page prefetching scheme for
disaggregated systems, which however can only help applications with high
locality within pages, and does not capture the significant variability in
data access costs of fully disaggregated systems. _DaeMon_ is orthogonal to
page prefetchers and can work synergistically with them to even further
improve performance, as described in Section 4.7. We leave the experimentation
of their synergy for future work.

Hybrid Memory Systems. Numerous works for hybrid memory systems propose data
placement schemes (Agarwal et al., 2015; Chou et al., 2017; Dulloor et al.,
2016; Ruan et al., 2020; Kim et al., 2021; Doudali et al., 2021; Liu et al.,
2019; Kannan et al., 2017; Weiner et al., 2022; Chang et al., 2021; Feeley et
al., 1995), or selection methods (Dong et al., 2010; Doudali et al., 2019; Yan
et al., 2019; Agarwal and Wenisch, 2017; Meswani et al., 2015; Liu et al.,
2017; Kotra et al., 2018; Jiang et al., 2010; Vasilakis et al., 2019;
Prodromou et al., 2017; Singh et al., 2022; Lagar-Cavilla et al., 2019) to
identify hot memory pages that are migrated to die-stacked DRAM, that is
organized as a cache of a larger main memory. Compared to these approaches,
first, intelligent page placement/movement is orthogonal to _DaeMon_ , and
cannot by itself address the high overheads caused by remote page migrations
across the network, that can be significantly slower than that within the
server and more latency/bandwidth-constrained in the context of fully
disaggregated systems. Second, these prior works assume a monolithic
centralized system where TLBs/page tables can be leveraged to track page
hotness of remote pages (e.g., (Dong et al., 2010; Doudali et al., 2019; Yan
et al., 2019; Liu et al., 2017; Kotra et al., 2018; Agarwal and Wenisch, 2017;
Meswani et al., 2015)) or that memory allocation/placement is handled by the
server itself (e.g., (Ruan et al., 2020; Dulloor et al., 2016; Agarwal et al.,
2015; Kim et al., 2021; Doudali et al., 2021; Liu et al., 2019; Kannan et al.,
2017; Li et al., 2017; Feeley et al., 1995)). However, in disaggregated
systems, address translation and memory management are distributed across
memory components and cannot be used to track pages at the CPU server side,
while compute components and memory components are managed by independent
kernel monitors that have no visibility/control of other components or data
management/placement across components. Similarly, hardware-based approaches
(Vasilakis et al., 2019; Jiang et al., 2010; Prodromou et al., 2017; Li et
al., 2017) for hybrid systems add _centralized_ hardware units at the server
side to store page tracking metadata for the second-tier main memory. For
example, Chop (Jiang et al., 2010) adds 4MB of metadata to track 16GB of
second-tier memory. These schemes would incur significant area overheads (in
the order of GBs) to track large amounts of remote memory (in the order of
TBs) enabled by disaggregated systems (Shan et al., 2018). Requiring _each_
compute component to track a large number of pages enabled by multiple remote
memory components would cause scalability issues and significantly limit the
benefits of resource disaggregation. Thus, designing an effective scalable hot
page selection scheme for fully disaggregated systems is an open challenge,
and _DaeMon_ could work in conjunction with such schemes to further improve
performance. Third, all these prior works do not handle variability in data
access costs of disaggregated systems. disaggregated systems necessitate an
adaptive mechanism given the significant variations in access latencies and
bandwidth. Fourth, applying/adapting the design of prior schemes tailored for
tightly-integrated hybrid systems in disaggregated systems might incur
significantly higher overheads and require important modifications than that
described in the original papers.

A few recent works design hardware schemes for commodity servers to enable
moving data _only_ at cache line granularity (Loh and Hill, 2011, 2012; Chou
et al., 2014; Vasilakis, Evangelos and Papaefstathiou, Vassilis and Trancoso,
Pedro and Sourdis, Ioannis, 2020) or a larger sub-block granularity (a few
cache lines) (Ryoo et al., 2017; Jevdjic et al., 2013). Ekman et al. (Ekman
and Stenstrom, 2005a) evaluate a critical-block first approach, where each 8KB
page is split in blocks of 2KB data, and the requested (critical) 2KB block of
data is transferred first, and written in DRAM cache. As we show in § 2.2,
moving data at a single granularity (page or cache line) can incur high
performance costs and does not provide robustness towards significant
variations in network bandwidth and latencies.

Hardware Compression. Prior works propose compression schemes (Pekhimenko et
al., 2012; Kim et al., 2016; Alameldeen and Wood, 2004b; Thuresson and
Stenström, 2008; Chen et al., 2010; Yang et al., 2000, 2004; Arelakis and
Stenstrom, 2014; Nguyen and Wentzlaff, 2015; Nguyen et al., 2018; Shafiee et
al., 2014; Young et al., 2019; Ekman and Stenstrom, 2005b; Pekhimenko et al.,
2013; Choukse et al., 2018; Qian et al., 2018; Wilson et al., 1999; Kjelso et
al., 1996; Tian et al., 2014; Park et al., 2021; Abali et al., 2001) for cache
memory, main memory and memory bus links in CPUs/GPUs (Mittal and Vetter,
2016; Thuresson et al., 2008; Sathish et al., 2012; Vijaykumar et al., 2015),
and selection methods to dynamically enable/disable compression (Alameldeen
and Wood, 2004a, 2007; Tuduce and Gross, 2005), or find the best-performing
compression scheme (Arelakis et al., 2015; Kim et al., 2017). These works
integrate ratio-optimized or latency-optimized compression schemes depending
on the particular context and system’s characteristics they target. Our work
enables link compression in page movements synergistically with decoupled
multiple granularity data movement, which allows us to tolerate the high
compression latencies of ratio-optimized compression schemes such as LZ (Ziv
and Lempel, 1977).

##  8\. Conclusion

_DaeMon_ is the first adaptive data movement solution for fully disaggregated
systems. _DaeMon_ supports low-cost page migration, scales elastically to
multiple hardware components, enables software transparency, and provides
robustness across various architecture/network characteristics and the
application behavior by effectively monitoring _pending_ cache line and page
movements. Our evaluations using a state-of-the-art simulator show that
_DaeMon_ significantly improves system performance and data access costs for a
wide range of applications under various architecture and network
configurations, and when multiple jobs are simultaneously running in the
system. We conclude that _DaeMon_ is an efficient, scalable and robust
solution to alleviate data movement overheads in disaggregated systems, and
hope that this work encourages further studies of the data movement problem in
disaggregated systems.

###### Acknowledgements.

We thank the anonymous reviewers from SIGMETRICS 2023, and our shepherd,
Abhishek Chandra, for their comments and suggestions. We also thank
Konstantinos Kanellopoulos and Ivan Fernandez for their help on technical
aspects of this work.

## References

  * (1)
  * Abali et al. (2001) B. Abali, H. Franke, D. E. Poff, R. A. Saccone, C. O. Schulz, L. M. Herger, and T. B. Smith, _Memory Expansion Technology (MXT): Software Support and Performance_ , _IBM Journal of Research and Development_ , 2001\.
  * Adya et al. (2019) Atul Adya, Robert Grandl, Daniel Myers, and Henry Qin, _Fast Key-Value Stores: An Idea Whose Time Has Come and Gone_ , _HotOS_ , 2019.
  * Agarwal et al. (2015) Neha Agarwal, David Nellans, Mark Stephenson, Mike O’Connor, and Stephen W. Keckler, _Page Placement Strategies for GPUs within Heterogeneous Memory Systems_ , _ASPLOS_ , 2015.
  * Agarwal and Wenisch (2017) Neha Agarwal and Thomas F. Wenisch, _Thermostat: Application-Transparent Page Management for Two-Tiered Main Memory_ , _ASPLOS_ , 2017.
  * Aguilera et al. (2018) Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Stanko Novaković, Arun Ramanathan, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei, _Remote Regions: A Simple Abstraction for Remote Memory_ , _ATC_ , 2018.
  * Aguilera et al. (2017) Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei, _Remote Memory in the Age of Fast Networks_ , _SoCC_ , 2017.
  * Alameldeen and Wood (2004a) A.R. Alameldeen and D.A. Wood, _Adaptive Cache Compression for High-Performance Processors_ , _ISCA_ , 2004\.
  * Alameldeen and Wood (2004b) Alaa Alameldeen and David Wood, _Frequent Pattern Compression: A Significance-Based Compression Scheme for L2 Caches_ , _Technical Report_ , 2004.
  * Alameldeen and Wood (2007) Alaa R. Alameldeen and David A. Wood, _Interactions Between Compression and Prefetching in Chip Multiprocessors_ , _HPCA_ , 2007.
  * Angel et al. (2020) Sebastian Angel, Mihir Nanavati, and Siddhartha Sen, _Disaggregation and the Application_ , _HotCloud_ , 2020.
  * Arelakis et al. (2015) Angelos Arelakis, Fredrik Dahlgren, and Per Stenstrom, _HyComp: A Hybrid Cache Compression Method for Selection of Data-Type-Specific Compression Methods_ , _MICRO_ , 2015.
  * Arelakis and Stenstrom (2014) Angelos Arelakis and Per Stenstrom, _SC2: A Statistical Compression Cache Scheme_ , _ISCA_ , 2014.
  * Assn. (2017) JEDEC Solid State Technology Assn., _JESD79-4B: DDR4 SDRAM Standard_ , _http://www.softnology.biz/pdf/JESD79-4B.pdf_ , 2017\.
  * Bindschaedler et al. (2020) Laurent Bindschaedler, Ashvin Goel, and Willy Zwaenepoel, _Hailstorm: Disaggregated Compute and Storage for Distributed LSM-Based Databases_ , _ASPLOS_ , 2020\.
  * Buragohain et al. (2017) Dhantu Buragohain, Abhishek Ghogare, Trishal Patel, Mythili Vutukuru, and Purushottam Kulkarni, _DiME: A Performance Emulator for Disaggregated Memory Architectures_ , _APSys_ , 2017\.
  * Calciu et al. (2021) Irina Calciu, M. Talha Imran, Ivan Puddu, Sanidhya Kashyap, Hasan Al Maruf, Onur Mutlu, and Aasheesh Kolli, _Rethinking Software Runtimes for Disaggregated Memory_ , _ASPLOS_ , 2021.
  * Carlson et al. (2011) Trevor E. Carlson, Wim Heirman, and Lieven Eeckhout, _Sniper: Exploring the Level of Abstraction for Scalable and Accurate Parallel Multi-Core Simulations_ , _SC_ , 2011.
  * Carlson et al. (2014) Trevor E. Carlson, Wim Heirman, Stijn Eyerman, Ibrahim Hur, and Lieven Eeckhout, _An Evaluation of High-Level Mechanistic Core Models_ , _TACO_ , 2014.
  * Chang et al. (2021) Chia-Hao Chang, Adithya Kumar, and Anand Sivasubramaniam, _To Move or Not to Move? Page Migration for Irregular Applications in over-Subscribed GPU Memory Systems with DynaMap_ , _SYSTOR_ , 2021.
  * Che et al. (2009) Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-Ha Lee, and Kevin Skadron, _Rodinia: A Benchmark Suite for Heterogeneous Computing_ , _IISWC_ , 2009.
  * Chen et al. (2010) Xi Chen, Lei Yang, Robert P. Dick, Li Shang, and Haris Lekatsas, _C-Pack: A High-Performance Microprocessor Cache Compression Algorithm_ , _VLSI_ , 2010.
  * Chou et al. (2017) Chiachen Chou, Aamer Jaleel, and Moinuddin Qureshi, _BATMAN: Techniques for Maximizing System Bandwidth of Memory Systems with Stacked-DRAM_ , _MEMSYS_ , 2017\.
  * Chou et al. (2014) Chia Chen Chou, Aamer Jaleel, and Moinuddin K. Qureshi, _CAMEO: A Two-Level Memory Organization with Capacity of Main Memory and Flexibility of Hardware-Managed Cache_ , _MICRO_ , 2014.
  * Choukse et al. (2018) Esha Choukse, Mattan Erez, and Alaa R. Alameldeen, _Compresso: Pragmatic Main Memory Compression_ , _MICRO_ , 2018.
  * Cock et al. (2022) David Cock, Abishek Ramdas, Daniel Schwyn, Michael Giardino, Adam Turowski, Zhenhao He, Nora Hossle, Dario Korolija, Melissa Licciardello, Kristina Martsenko, Reto Achermann, Gustavo Alonso, and Timothy Roscoe, _Enzian: An Open, General, CPU/FPGA Platform for Systems Software Research_ , _ASPLOS_ , 2022.
  * Dong et al. (2010) Xiangyu Dong, Yuan Xie, Naveen Muralimanohar, and Norman P. Jouppi, _Simple but Effective Heterogeneous Main Memory with On-Chip Memory Controller Support_ , _SC_ , 2010.
  * Doudali et al. (2019) Thaleia Dimitra Doudali, Sergey Blagodurov, Abhinav Vishnu, Sudhanva Gurumurthi, and Ada Gavrilovska, _Kleio: A Hybrid Memory Page Scheduler with Machine Intelligence_ , _HPDC_ , 2019\.
  * Doudali et al. (2021) Thaleia Dimitra Doudali, Daniel Zahka, and Ada Gavrilovska, _Cori: Dancing to the Right Beat of Periodic Data Movements over Hybrid Memory Systems_ , _IPDPS_ , 2021\.
  * Dulloor et al. (2016) Subramanya R. Dulloor, Amitabha Roy, Zheguang Zhao, Narayanan Sundaram, Nadathur Satish, Rajesh Sankaran, Jeff Jackson, and Karsten Schwan, _Data Tiering in Heterogeneous Memory Systems_ , _EuroSys_ , 2016\.
  * Ekman and Stenstrom (2005a) Magnus Ekman and Per Stenstrom, _A Cost-Effective Main Memory Organization for Future Servers_ , _IPDPS_ , 2005\.
  * Ekman and Stenstrom (2005b) M. Ekman and P. Stenstrom, _A Robust Main-Memory Compression Scheme_ , _ISCA_ , 2005.
  * Feeley et al. (1995) M. J. Feeley, W. E. Morgan, E. P. Pighin, A. R. Karlin, H. M. Levy, and C. A. Thekkath, _Implementing Global Memory Management in a Workstation Cluster_ , _SOSP_ , 1995.
  * Gao et al. (2016) Peter X. Gao, Akshay Narayan, Sagar Karandikar, Joao Carreira, Sangjin Han, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker, _Network Requirements for Resource Disaggregation_ , _OSDI_ , 2016.
  * Giannoula (2022) Christina Giannoula, _Accelerating Irregular Applications via Efficient Synchronization and Data Access Techniques_ , _https://arxiv.org/abs/2211.05908_ , 2022\.
  * Gouk et al. (2022) Donghyun Gouk, Sangwon Lee, Miryeong Kwon, and Myoungsoo Jung, _Direct Access, High-Performance Memory Disaggregation with DirectCXL_ , _ATC_ , 2022\.
  * Gu et al. (2017) Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin, _Efficient Memory Disaggregation with Infiniswap_ , _NSDI_ , 2017\.
  * Guo et al. (2016) Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn, _RDMA over Commodity Ethernet at Scale_ , _SIGCOMM_ , 2016\.
  * Guo et al. (2022) Zhiyuan Guo, Yizhou Shan, Xuhao Luo, Yutong Huang, and Yiying Zhang, _Clio: A Hardware-Software Co-Designed Disaggregated Memory System_ , _ASPLOS_ , 2022.
  * Han et al. (2013) Sangjin Han, Norbert Egi, Aurojit Panda, Sylvia Ratnasamy, Guangyu Shi, and Scott Shenker, _Network Support for Resource Disaggregation in Next-Generation Datacenters_ , _HotNets_ , 2013\.
  * HPCG (2019) HPCG, _High Performance Conjugate Gradient Benchmark_ , _https://github.com/hpcg-benchmark/hpcg_ , 2019\.
  * Hwang et al. (2020) Ranggi Hwang, Taehun Kim, Youngeun Kwon, and Minsoo Rhu, _Centaur: A Chiplet-Based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations_ , _ISCA_ , 2020.
  * Intel (2021) Intel, _Intel Omni-Path Architecture_ , _https://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-driving-exascale-computing.html_ , 2021\.
  * Jevdjic et al. (2013) Djordje Jevdjic, Stavros Volos, and Babak Falsafi, _Die-Stacked DRAM Caches for Servers: Hit Ratio, Latency, or Bandwidth? Have It All with Footprint Cache_ , _ISCA_ , 2013.
  * Jiang et al. (2010) Xiaowei Jiang, Niti Madan, Li Zhao, Mike Upton, Ravishankar Iyer, Srihari Makineni, Donald Newell, Yan Solihin, and Rajeev Balasubramonian, _CHOP: Adaptive Filter-Based DRAM Caching for CMP Server Platforms_ , _HPCA_ , 2010.
  * Jun et al. (2017) Hongshin Jun, Jinhee Cho, Kangseol Lee, Ho-Young Son, Kwiwook Kim, Hanho Jin, and Keith Kim, _HBM DRAM Technology and Architecture_ , _IMW_ , 2017.
  * Kannan et al. (2017) Sudarsun Kannan, Ada Gavrilovska, Vishal Gupta, and Karsten Schwan, _HeteroOS: OS Design for Heterogeneous Memory Management in Datacenter_ , _ISCA_ , 2017\.
  * Katrinis et al. (2016) K. Katrinis, D. Syrivelis, D. Pnevmatikatos, G. Zervas, D. Theodoropoulos, I. Koutsopoulos, K. Hasharoni, D. Raho, C. Pinto, F. Espina, S. Lopez-Buedo, Q. Chen, M. Nemirovsky, D. Roca, H. Klos, and T. Berends, _Rack-Scale Disaggregated Cloud Data Centers: The dReDBox Project Vision_ , _DATE_ , 2016.
  * Kim et al. (2021) Jonghyeon Kim, Wonkyo Choe, and Jeongseob Ahn, _Exploring the Design Space of Page Management for Multi-Tiered Memory Systems_ , _ATC_ , 2021\.
  * Kim et al. (2016) Jungrae Kim, Michael Sullivan, Esha Choukse, and Mattan Erez, _Bit-Plane Compression: Transforming Data for Better Compression in Many-Core Architectures_ , _ISCA_ , 2016.
  * Kim et al. (2017) Seikwon Kim, Seonyoung Lee, Taehoon Kim, and Jaehyuk Huh, _Transparent Dual Memory Compression Architecture_ , _PACT_ , 2017.
  * Kjelso et al. (1996) M. Kjelso, M. Gooch, and S. Jones, _Design and Performance of a Main Memory Hardware Data Compressor_ , _EUROMICRO_ , 1996.
  * Kjolstad et al. (2017) Fredrik Kjolstad, Stephen Chou, David Lugato, Shoaib Kamil, and Saman Amarasinghe, _Taco: A Tool to Generate Tensor Algebra Kernels_ , _ASE_ , 2017\.
  * Kotra et al. (2018) Jagadish B. Kotra, Haibo Zhang, Alaa R. Alameldeen, Chris Wilkerson, and Mahmut T. Kandemir, _CHAMELEON: A Dynamically Reconfigurable Heterogeneous Memory System_ , _MICRO_ , 2018\.
  * Lagar-Cavilla et al. (2019) Andres Lagar-Cavilla, Junwhan Ahn, Suleiman Souhlal, Neha Agarwal, Radoslaw Burny, Shakeel Butt, Jichuan Chang, Ashwin Chaugule, Nan Deng, Junaid Shahid, Greg Thelen, Kamil Adam Yurtsever, Yu Zhao, and Parthasarathy Ranganathan, _Software-Defined Far Memory in Warehouse-Scale Computers_ , _ASPLOS_ , 2019.
  * Lee et al. (2021) Seung-seob Lee, Yanpeng Yu, Yupeng Tang, Anurag Khandelwal, Lin Zhong, and Abhishek Bhattacharjee, _MIND: In-Network Memory Management for Disaggregated Data Centers_ , _SOSP_ , 2021\.
  * Li et al. (2017) Yang Li, Saugata Ghose, Jongmoo Choi, Jin Sun, Hui Wang, and Onur Mutlu, _Utility-Based Hybrid Memory Management_ , _CLUSTER_ , 2017.
  * Lim et al. (2009) Kevin Lim, Jichuan Chang, Trevor Mudge, Parthasarathy Ranganathan, Steven K. Reinhardt, and Thomas F. Wenisch, _Disaggregated Memory for Expansion and Sharing in Blade Servers_ , _ISCA_ , 2009\.
  * Lim et al. (2012) Kevin Lim, Yoshio Turner, Jose Renato Santos, Alvin AuYoung, Jichuan Chang, Parthasarathy Ranganathan, and Thomas F. Wenisch, _System-Level Implications of Disaggregated Memory_ , _HPCA_ , 2012.
  * Liu et al. (2017) Haikun Liu, Yujie Chen, Xiaofei Liao, Hai Jin, Bingsheng He, Long Zheng, and Rentong Guo, _Hardware/Software Cooperative Caching for Hybrid DRAM/NVM Memory Architectures_ , _ICS_ , 2017.
  * Liu et al. (2019) Lei Liu, Shengjie Yang, Lu Peng, and Xinyu Li, _Hierarchical Hybrid Memory Management in OS for Tiered Memory Systems_ , _TPDS_ , 2019\.
  * Loh and Hill (2012) Gabriel Loh and Mark D. Hill, _Supporting Very Large DRAM Caches with Compound-Access Scheduling and MissMap_ , _IEEE Micro_ , 2012.
  * Loh and Hill (2011) Gabriel H. Loh and Mark D. Hill, _Efficiently Enabling Conventional Block Sizes for Very Large Die-Stacked DRAM Caches_ , _MICRO_ , 2011.
  * Maruf and Chowdhury (2020) Hasan Al Maruf and Mosharaf Chowdhury, _Effectively Prefetching Remote Memory with Leap_ , _ATC_ , 2020\.
  * Mellanox (2020) Mellanox, _Mellanox Innova Adapters._ , _https://www.nvidia.com/en-us/networking/products/data-processing-unit/?mtag=programmable_adapter_cards_ , 2020\.
  * Meswani et al. (2015) Mitesh R. Meswani, Sergey Blagodurov, David Roberts, John Slice, Mike Ignatowski, and Gabriel H. Loh, _Heterogeneous Memory Architectures: A HW/SW Approach for Mixing Die-Stacked and Off-Package Memories_ , _HPCA_ , 2015.
  * Mittal and Vetter (2016) Sparsh Mittal and Jeffrey S. Vetter, _A Survey Of Architectural Approaches for Data Compression in Cache and Main Memory Systems_ , _TPDS_ , 2016.
  * Muralimanohar et al. (2007) Naveen Muralimanohar, Rajeev Balasubramonian, and Norm Jouppi, _Optimizing NUCA Organizations and Wiring Alternatives for Large Caches with CACTI 6.0_ , _MICRO_ , 2007.
  * Naumov et al. (2019) Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy, _Deep Learning Recommendation Model for Personalization and Recommendation Systems_ , _arXiv_ , 2019.
  * Nguyen et al. (2018) Tri M. Nguyen, Adi Fuchs, and David Wentzlaff, _CABLE: A CAche-Based Link Encoder for Bandwidth-Starved Manycores_ , _MICRO_ , 2018.
  * Nguyen and Wentzlaff (2015) Tri M. Nguyen and David Wentzlaff, _MORC: A Manycore-Oriented Compressed Cache_ , _MICRO_ , 2015.
  * Nitu et al. (2018) Vlad Nitu, Boris Teabe, Alain Tchana, Canturk Isci, and Daniel Hagimont, _Welcome to Zombieland: Practical and Energy-Efficient Memory Disaggregation in a Datacenter_ , _EuroSys_ , 2018.
  * Park et al. (2021) Sungbo Park, Ingab Kang, Yaebin Moon, Jung Ho Ahn, and G. Edward Suh, _BCD Deduplication: Effective Memory Compression Using Partial Cache-Line Deduplication_ , _ASPLOS_ , 2021.
  * Pekhimenko et al. (2013) Gennady Pekhimenko, Vivek Seshadri, Yoongu Kim, Hongyi Xin, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, and Todd C. Mowry, _Linearly Compressed Pages: A Low-Complexity, Low-Latency Main Memory Compression Framework_ , _MICRO_ , 2013.
  * Pekhimenko et al. (2012) Gennady Pekhimenko, Vivek Seshadri, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, and Todd C. Mowry, _Base-Delta-Immediate Compression: Practical Data Compression for on-Chip Caches_ , _PACT_ , 2012.
  * Peng et al. (2020) Ivy Peng, Roger Pearce, and Maya Gokhale, _On the Memory Underutilization: Exploring Disaggregated Memory on HPC Systems_ , _SBAC-PAD_ , 2020.
  * Pinto et al. (2020) Christian Pinto, Dimitris Syrivelis, Michele Gazzetti, Panos Koutsovasilis, Andrea Reale, Kostas Katrinis, and H. Peter Hofstee, _ThymesisFlow: A Software-Defined, HW/SW co-Designed Interconnect Stack for Rack-Scale Memory Disaggregation_ , _MICRO_ , 2020.
  * Prodromou et al. (2017) Andreas Prodromou, Mitesh Meswani, Nuwan Jayasena, Gabriel Loh, and Dean M. Tullsen, _MemPod: A Clustered Architecture for Efficient and Scalable Migration in Flat Address Space Multi-level Memories_ , _HPCA_ , 2017\.
  * Qian et al. (2018) Cheng Qian, Libo Huang, Qi Yu, Zhiying Wang, and Bruce Childers, _CMH: Compression Management for Improving Capacity in the Hybrid Memory Cube_ , _CF_ , 2018.
  * Rao and Porter (2016) Pramod Subba Rao and George Porter, _Is Memory Disaggregation Feasible? A Case Study with Spark SQL_ , _ANCS_ , 2016.
  * RDMA (2019) RDMA, _RDMA Consortium_ , _http://www.rdmaconsortium.org/_ , 2019\.
  * RDMA (2022) RDMA, _Gen-Z Core Specification_ , _https://genzconsortium.org/_ , 2022\.
  * Redmon (2016) Joseph Redmon, _Darknet: Open Source Neural Networks in C_ , _http://pjreddie.com/darknet/_ , 2013–2016.
  * Ruan et al. (2020) Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay, _AIFM: High-Performance, Application-Integrated Far Memory_ , _OSDI_ , 2020.
  * Ryoo et al. (2017) Jee Ho Ryoo, Mitesh R. Meswani, Andreas Prodromou, and Lizy K. John, _SILC-FM: Subblocked InterLeaved Cache-Like Flat Memory Organization_ , _HPCA_ , 2017\.
  * Sapio et al. (2017) Amedeo Sapio, Ibrahim Abdelaziz, Abdulla Aldilaijan, Marco Canini, and Panos Kalnis, _In-Network Computation is a Dumb Idea Whose Time Has Come_ , _HotNets_ , 2017.
  * Sathish et al. (2012) Vijay Sathish, Michael J. Schulte, and Nam Sung Kim, _Lossless and Lossy Memory I/O Link Compression for Improving Performance of GPGPU Workloads_ , _PACT_ , 2012\.
  * Shafiee et al. (2014) Ali Shafiee, Meysam Taassori, Rajeev Balasubramonian, and Al Davis, _MemZip: Exploring Unconventional Benefits from Memory Compression_ , _HPCA_ , 2014.
  * Shan et al. (2018) Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang, _LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation_ , _OSDI_ , 2018\.
  * Shun and Blelloch (2013) Julian Shun and Guy E. Blelloch, _Ligra: A Lightweight Graph Processing Framework for Shared Memory_ , _PpopP_ , 2013\.
  * Singh et al. (2022) Gagandeep Singh, Rakesh Nadig, Jisung Park, Rahul Bera, Nastaran Hajinazar, David Novo, Juan Gómez-Luna, Sander Stuijk, Henk Corporaal, and Onur Mutlu, _Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learning_ , _ISCA_ , 2022.
  * Thuresson et al. (2008) Martin Thuresson, Lawrence Spracklen, and Per Stenstrom, _Memory-Link Compression Schemes: A Value Locality Perspective_ , _IEEE Trans. Comput._ , 2008\.
  * Thuresson and Stenström (2008) Martin Thuresson and Per Stenström, _Accommodation of the Bandwidth of Large Cache Blocks Using Cache/Memory Link Compression_ , _ICPP_ , 2008.
  * Tian et al. (2014) Yingying Tian, Samira M. Khan, Daniel A. Jiménez, and Gabriel H. Loh, _Last-Level Cache Deduplication_ , _ICS_ , 2014.
  * Tremaine et al. (2001) R.B. Tremaine, T.B. Smith, M. Wazlowski, D. Har, Kwok-Ken Mak, and S. Arramreddy, _Pinnacle: IBM MXT in a Memory Controller Chip_ , _IEEE Micro_ , 2001.
  * Tsai and Zhang (2017) Shin-Yeh Tsai and Yiying Zhang, _LITE Kernel RDMA Support for Datacenter Applications_ , _SOSP_ , 2017.
  * Tuduce and Gross (2005) Irina Chihaia Tuduce and Thomas Gross, _Adaptive Main Memory Compression_ , _ATC_ , 2005.
  * Vasilakis et al. (2019) Evangelos Vasilakis, Vassilis Papaefstathiou, Pedro Trancoso, and Ioannis Sourdis, _LLC-Guided Data Migration in Hybrid Memory Systems_ , _IPDPS_ , 2019.
  * Vasilakis, Evangelos and Papaefstathiou, Vassilis and Trancoso, Pedro and Sourdis, Ioannis (2020) Vasilakis, Evangelos and Papaefstathiou, Vassilis and Trancoso, Pedro and Sourdis, Ioannis, _Hybrid2: Combining Caching and Migration in Hybrid Memory Systems_ , _HPCA_ , 2020.
  * Vijaykumar et al. (2015) Nandita Vijaykumar, Gennady Pekhimenko, Adwait Jog, Abhishek Bhowmick, Rachata Ausavarungnirun, Chita Das, Mahmut Kandemir, Todd C. Mowry, and Onur Mutlu, _A Case for Core-Assisted Bottleneck Acceleration in GPUs: Enabling Flexible Data Compression with Assist Warps_ , _ISCA_ , 2015.
  * Wang et al. (2020) Chenxi Wang, Haoran Ma, Shi Liu, Yuanqi Li, Zhenyuan Ruan, Khanh Nguyen, Michael D. Bond, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu, _Semeru: A Memory-Disaggregated Managed Runtime_ , _OSDI_ , 2020.
  * Weiner et al. (2022) Johannes Weiner, Niket Agarwal, Dan Schatzberg, Leon Yang, Hao Wang, Blaise Sanouillet, Bikash Sharma, Tejun Heo, Mayank Jain, Chunqiang Tang, and Dimitrios Skarlatos, _TMO: Transparent Memory Offloading in Datacenters_ , _ASPLOS_ , 2022.
  * Wilson et al. (1999) P. Wilson, Scott F. Kaplan, and Y. Smaragdakis, _The Case for Compressed Caching in Virtual Memory Systems_ , _ATC_ , 1999.
  * Woo et al. (2010) Dong Hyuk Woo, Nak Hee Seong, Dean L. Lewis, and Hsien-Hsin Sean Lee, _An Optimized 3D-stacked Memory Architecture by Exploiting Excessive, High-Density TSV Bandwidth_ , _HPCA_ , 2010\.
  * Yan et al. (2019) Zi Yan, Daniel Lustig, David Nellans, and Abhishek Bhattacharjee, _Nimble Page Management for Tiered Memory Systems_ , _ASPLOS_ , 2019.
  * Yang et al. (2004) Jun Yang, Rajiv Gupta, and Chuanjun Zhang, _Frequent Value Encoding for Low Power Data Buses_ , _TODAES_ , 2004\.
  * Yang et al. (2000) Jun Yang, Youtao Zhang, and R. Gupta, _Frequent Value Compression in Data Caches_ , _MICRO_ , 2000\.
  * Yeh et al. (2016) Chin-Chia Michael Yeh, Yan Zhu, Liudmila Ulanova, Nurjahan Begum, Yifei Ding, Hoang Anh Dau, Diego Furtado Silva, Abdullah Mueen, and Eamonn Keogh, _Matrix Profile I: All Pairs Similarity Joins for Time Series: A Unifying View that Includes Motifs, Discords and Shapelets_ , _ICDM_ , 2016\.
  * Young et al. (2019) Vinson Young, Sanjay Kariyappa, and Moinuddin K. Qureshi, _Enabling Transparent Memory-Compression for Commodity Memory Systems_ , _HPCA_ , 2019\.
  * Zervas et al. (2018) Georgios Zervas, Hui Yuan, Arsalan Saljoghei, Qianqiao Chen, and Vaibhawa Mishra, _Optically Disaggregated Data Centers with Minimal Remote Memory Latency: Technologies, Architectures, and Resource Allocation_ , _JOCN_ , 2018.
  * Zhang et al. ([n. d.]) Qizhen Zhang, Yifan Cai, Sebastian G. Angel, Vincent Liu, Ang Chen, and B. T. Loo, _Rethinking Data Management Systems for Disaggregated Data Centers_ , _CIDR_ , [n. d.].
  * Zhou et al. (2022) Yang Zhou, Hassan M. G. Wassel, Sihang Liu, Jiaqi Gao, James Mickens, Minlan Yu, Chris Kennelly, Paul Turner, David E. Culler, Henry M. Levy, and Amin Vahdat, _Carbink: Fault-Tolerant Far Memory_ , _OSDI_ , 2022\.
  * Ziv and Lempel (1977) J. Ziv and A. Lempel, _A Universal Algorithm for Sequential Data Compression_ , _IEEE Transactions on Information Theory_ , 1977.
  * Zuo et al. (2021) Pengfei Zuo, Jiazhao Sun, Liu Yang, Shuangwu Zhang, and Yu Hua, _One-sided RDMA-Conscious Extendible Hashing for Disaggregated Memory_ , _ATC_ , 2021.

## APPENDIX

##  Appendix A Extended Results

###  A.1. Network Bandwidth Utilization

Figure 19 compares the bandwidth utilization across the network of a compute
component and a memory component achieved by various data movement schemes.

Figure 19. Bandwidth utilization (%) across the network of a compute component
and a memory component achieved by various data movement schemes.

We make three key observations. First, LC typically reduces the network
bandwidth utilization over Remote (by 2.49$\times$ on average across all
workloads and network configurations), because fewer bytes are transferred
through the network, since remote pages are migrated in a compressed format.
Note that LC improves the total execution time over Remote, and thus in a few
workloads, e.g., _pr_ , the network bandwidth utilization might be higher
within a smaller execution time. Second, PQ decreases the network bandwidth
utilization over Remote in workloads with poor spatial locality within pages
(e.g., _nw_), since the selection granularity unit effectively schedules more
cache line movements and fewer page migrations. Instead, PQ might slightly
increase the network bandwidth utilization over Remote in workloads with
medium spatial locality within pages (e.g., _bf_ , _bc_), since the selection
granularity unit enables both cache line and page migrations to leverage both
the ability to prioritize critical cache line requests and the benefits of
data locality within pages. In workloads with high spatial locality within
pages (e.g., _dr_ , _rs_), PQ favors more page migrations and fewer cache line
movements, thus achieving similar network bandwidth utilization to Remote.
Third, _DaeMon_ greatly decreases the network bandwidth utilization over
Remote by 2.32$\times$ on average across all workloads and network
configurations (not graphed). _DaeMon_ effectively transfers remote pages in a
compressed format and on-the-fly selects the granularity of data migrations to
_significantly_ reduce the bandwidth consumption across the network of fully
disaggregated systems.

###  A.2. Sensitivity Study to Switch Latency

Figure 20 compares _DaeMon_ ’s performance over Remote’s performance averaged
across all workloads, when varying the switch latency of the network. When the
fixed switch latency becomes very high dominating the total data movement
costs, _DaeMon_ has lower benefits over Remote, since _DaeMon_ does not hide
the propagation and switching delays in network components (e.g., fixed
processing costs of the packet inside network switches). However, even with a
very high switch latency in the order of microsecond, i.e., 1$\mu$s (=1000
ns), _DaeMon_ outperforms Remote by 1.49$\times$ on average across all
workloads.

Figure 20. Performance benefits of _DaeMon_ over Remote, when varying the
switch latency of the network.

###  A.3. Sensitivity Study to Network Bandwidth

To evaluate bandwidth-limited scenarios, Figure 21 compares _DaeMon_ ’s
performance normalized to Remote’s performance in multithreaded workloads
running on 8 OoO cores of a compute component, when varying the bandwidth
factor of the network, e.g., up to having a very low bandwidth factor of 1/16
(i.e., network bandwidth is 16$\times$ slower than the DRAM bus bandwidth)
between a compute component and memory component. We find that on average
_DaeMon_ ’s benefits increase over the widely-adopted approach of moving data
at page granularity, i.e., Remote, since _DaeMon_ even more significantly
alleviates bandwidth bottlenecks and data movement overheads under bandwidth-
constrained scenarios.

Figure 21. Performance benefits of _DaeMon_ normalized to Remote using
multithreaded workloads, when varying the bandwidth factor of the network
between a compute component and memory component.

###  A.4. Performance Benefits With Multiple Memory Components

Figure 22 evaluates the performance of _DaeMon_ normalized to Remote’s
performance, when increasing the number of memory components in the system
having the same network configuration for each memory component, i.e., 100 ns
switch latency and a bandwidth factor of 1/4. We evaluated distributing memory
pages with either a round-robin way or randomly across multiple remote memory
components, and drew the same key observations for both distributions.
Similarly to Figure 17, we observe that when pages are distributed across
multiple memory components and the system provides larger aggregate network
and memory bandwidth, data access costs decrease. For example, when increasing
the number of memory components from 2 to 4, the remote data access latency
decreases by 1.39$\times$ on average across all workloads. However, even when
data access costs affect less the total execution time of applications,
_DaeMon_ still further mitigates data access overheads: _DaeMon_ outperforms
the widely-adopted Remote approach by 2.09$\times$ and 1.88$\times$ on average
across all workloads, when using 2 and 4 memory components, respectively.

Figure 22. Performance benefits of _DaeMon_ normalized to Remote, when
increasing the number of memory components having 100 ns switch latency and a
bandwidth factor of 1/4 for each memory component.