# Dynamic Page Placement on Real Persistent Memory Systems

Miguel Marques, Ilia Kuzmin, João Barreto, José Monteiro, and Rodrigo
Rodrigues
INESC-ID, Instituto Superior Técnico, Universidade de Lisboa
{miguel.soares.marques, ilia.kuzmin<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As persistent memory (PM) technologies emerge, hybrid memory architectures
combining DRAM with PM bring the potential to provide a tiered, byte-
addressable main memory of unprecedented capacity. Nearly a decade after the
first proposals for these hybrid architectures, the real technology has
finally reached commercial availability with Intel Optane™ DC Persistent
Memory (DCPMM). This raises the challenge of designing systems that realize
this potential in practice, namely through effective approaches that
dynamically decide at which memory tier should pages be placed.

In this paper, we are the first, to our knowledge, to systematically analyze
tiered page placement on real DCPMM-based systems. To this end, we start by
revisiting the assumptions of state-of-the-art proposals, and confronting them
with the idiosyncrasies of today’s off-the-shelf DCPMM-equipped architectures.
This empirical study reveals that some of the key design choices in the
literature rely on important assumptions that are not verified in present-day
DRAM-DCPMM memory architectures.

Based on the lessons from this study, we design and implement HyPlacer, a tool
for tiered page placement in _off-the-shelf_ Linux-based systems equipped with
DRAM+DCPMM. In contrast to previous proposals, HyPlacer follows an approach
guided by two main practicality principles: 1) it is tailored to the
performance idiosyncrasies of off-the-shelf DRAM+DCPMM systems; and 2) it can
be seamlessly integrated into Linux with minimal kernel-mode components, while
ensuring extensibility to other HMAs and other data placement policies. Our
experimental evaluation of HyPlacer shows that it outperforms both solutions
proposed in past literature and placement options that are currently available
in off-the-shelf DCPMM-equipped Linux systems, reaching an improvement of up
to 11x when compared to the default memory policy in Linux.

##  1 Introduction

Today’s data-intensive applications are characterized by a high degree of data
complexity and parallelism, as well as an ever-increasing demand for memory
capacity. However, scaling up DRAM memory to address such trends is rapidly
hitting a well-known wall – the amount of available DRAM is limited by power,
cooling, area constraints, and cost [31].

In this context, the emerging persistent memory (PM) technologies can become a
game changer, not so much for their most highlighted property – persistence –,
but for an unprecedented combination of other properties: PM delivers
read/write latencies in the nanosecond range; it is byte-addressable, allowing
applications to directly access them by issuing load and store instructions;
and, compared to DRAM, PM is denser, providing a higher capacity per module,
while offering lower cost per bit.

As a consequence, a natural trend is to have heterogeneous memory hierarchies
(HMHs), which combine both memory technologies, PM and DRAM. This enables a
_tiered_ main memory, where applications can place and process much larger
working sets than in DRAM-only systems, therefore minimizing accesses to local
or distributed storage. Such HMHs are useful even for volatile working sets,
which are the focus of our paper.

The performance duality in DRAM+PM HMHs raises a challenging data placement
problem: _at which tier in the HMH should the data objects of an application
be placed for optimal performance?_ Allocating data to the appropriate memory,
taking into account the performance differences between both tiers, becomes a
decisive challenge to the effective scalability of data-intensive
applications.

These data placement strategies received considerable attention from the
research community in the last decade, following the initial promises of new
PM technologies and their integration into DRAM+PM HMHs. However, the
proposals that resulted from these initial research efforts suffer from the
intrinsic limitation of not having access to commercial PM hardware, since it
was not available at the time. In particular, we identify two fundamental
shortcomings in this initial research.

First, these proposals are designed based on conjectural assumptions about the
performance of then future PM-based systems. Furthermore, their experimental
evaluation relies on simulation or emulation methods that fail to capture
performance intricacies that are only noticeable with the real technology
[14].

Second, with a few exceptions, the early research work generally overlooks the
issue of how the proposed techniques are to be integrated into the system
stack of PM-based systems. As we detail later on, some papers simply propose
placement algorithms, regardless of how they can be implemented within the
complex system stack of today’s systems; other proposals resort to _ad hoc_
hardware extensions and, as such, they cannot be used with off-the-shelf PM-
based systems.

After a decade of research based on performance models that could not be
validated, PM has finally reached commercial availability with Intel Optane™
Data Center Persistent Memory modules (DCPMM). This constitutes a notable
opportunity to revisit the research avenue paved by previous proposals, now
with the ambitious goal of devising practical proposals whose design,
implementation and evaluation is no longer haunted by the above-mentioned
limitations.

This paper is the first, to our knowledge, to address that goal.

As a first contribution, we revisit the design space of a large body of pre-
DCPMM state-of-the-art proposals and confront them with the idiosyncrasies of
today’s off-the-shelf DCPMM-equipped architectures. This empirical study
reveals that some key design choices in the literature rely on fundamental
assumptions that are not verified in contemporary DRAM-DCPMM HMHs. On one
extreme, we find that simplistic placement policies that place read-only pages
in PM and read-write pages in DRAM can be highly inefficient when used with
DCPMM. On the other extreme, we find that sophisticated placement policies
that balance pages across PM and DRAM to maximize the aggregate access
bandwidth, when used with DCPMM, cannot achieve the gains that were foreseen
in pre-DCPMM literature. In-between both extremes, we focus on the _fill DRAM
first_ placement policy and show that, to increase its potential advantages
with DCPMM, both per-page hotness and read-write intensity should be monitored
and taken into account to guide the policy.

From this study, we devise a set of design guidelines for tiered page
placement that take into account the lessons learned from this first part.

As a second contribution, we propose HyPlacer, a practical tool for tiered
page placement in _off-the-shelf_ Linux-based systems equipped with DRAM+DCPMM
HMHs. In contrast to previous proposals, the design and implementation of
HyPlacer follow a pragmatic approach that, from the ground up, is guided by
two main practicality principles: 1) to be tailored to the performance
idiosyncrasies of off-the-shelf DRAM+DCPMM systems; and 2) to fit seamlessly
into Linux with minimal kernel-mode components.

In a nutshell, HyPlacer considers the disparity in performance between DRAM
and DCPMM, and dynamically decides new page distributions that ultimately lead
to a higher application throughput. In order to achieve this, HyPlacer
periodically identifies frequently modified and referenced pages, which
benefit the most from the fast, but limited, DRAM tier. As a result, a larger
portion of the workload’s accesses is fulfilled by the DRAM tier, while DCPMM
serves sporadically-read pages. We focus our efforts on a solution that
requires minimal changes to the Linux kernel, and expands existing page
placement mechanisms in order to accommodate the integration of DCPMM.
HyPlacer is implemented into a contemporary Linux system by changing only 1
line in Linux’ kernel code.

As a third contribution, we present a comprehensive experimental evaluation of
state-of-the-art proposals for tiered page placement, with realistic data-
intensive workloads from NAS Parallel Benchmark (NPB) [2] and GAP suites [4].
To our knowledge, this is the first experimental study to evaluate previous
proposals on a real system equipped with DCPMM memory. Our results show that,
in general, HyPlacer substantially outperforms previous state-of-the-art
solutions with an average speedup of 4.6x in large footprint workloads. When
compared to today’s Linux default policy for tiered page placement in
DRAM+DCPMM systems, HyPlacer achieves speedups of up to 11x. Considerable
gains are also found in energy consumption.

The remainder of this paper is organized as follows. Section Section 2
provides background on DCPMM and data placement efforts proposed in past
literature. Section 3 studies the performance of a real DCPMM-equipped system,
providing insights on how page placement should be attuned to such systems.
Section 4 describes HyPlacer, detailing its design and implementation choices.
Section 5 evaluates HyPlacer against existing tiered page placement
alternatives. Finally, Section 6 draws conclusions.

##  2 Background

In this section, we provide background on tiered page placement for DRAM+DCPMM
systems. The section is organized in a bottom-up fashion. In Section 2.1, we
start by describing DCPMM technology and how it is integrated into
contemporary architectures. Then, in Section 2.2 we explain how DCPMM is
exposed to the system software stack, and how the Linux operating system (OS)
currently supports DCPMM. Finally, in Section 2.3 we survey existing proposals
to data placement in heterogeneous memory systems, starting by a broad
perspective and then focusing on DCPMM.

T he main goal of our paper is to efficiently enable applications to place
large _volatile_ data sets in DRAM+DCPMM HMHs. Therefore, our background is
restricted to this scope. As such, our presentation does not cover the use of
DCPMM and other PM technologies to store crash-consistent persistent data
structures, which is a complementary research avenue that has been surveyed in
other papers [3, 41].

###  2.1 DCPMM internals

DCPMM is delivered as DIMMs that are compatible with DDR4 sockets. The current
capacity of DCPMM modules ranges from 128GB to 512GB, which represents up to a
4x increase in per-module capacity compared to DDR4 DRAM. Currently, DCPMM
modules can be used with 2nd or 3rd-generation Intel Xeon Scalable (Cascade
Lake) CPUs.

Each CPU contains 2 integrated memory controllers (iMC), with 3 memory
channels per iMC. Each memory channel can have up to 2 DIMM slots (depending
on the chosen motherboard), with the restriction that at most one DCPMM DIMM
may be installed at a given channel. Therefore, a given memory channel can
serve DRAM (up to 2 DIMMs), DCPMM (at most 1 DIMM) or a combination of 1 DRAM
+ 1 DCPMM DIMMs [18]. This flexibility enables these systems to have
DRAM+DCPMM HMHs with varying DRAM-DCPMM capacity and bandwidth ratios.

Each iMC uses the DDR-T protocol to communicate with DCPMM. Like DDR4, DDR-T
operates at cache-line granularity (usually 64B). Internally, each DCPMM
module caches 256B blocks (called XPLines), with an associated prefetcher
[39]. This cache also serves as a write-combining buffer for adjacent stores.
Due to the granularity mismatch between DDR4 and XPLines, random stores incur
in costly read-modify-write cycles [14]. Similar to SSDs, DCPMM uses logical
addressing for minimizing wear-leveling, leveraging an internal address
indirection table [39].

(a) ADM Configuration

(b) MemM Configuration

Figure 1: Main operation modes of DCPMM.

###  2.2 DCPMM operation modes and OS support

DCPMM can be configured in two different modes: _App Direct Mode_ (ADM), and
_Memory Mode_ (MemM), which Figure 1 illustrates in a single-socket
architecture. For simplicity, in the following we focus on a single-socket
scenario.

In the MemM configuration, the DCPMM memory is transparently accessible to the
OS as a single memory node, whose capacity is the same as the total DCPMM
capacity installed in the socket. DRAM is used as an internal last-level
cache, which interposes every access to the local DCPMM memory node.

When DCPMM is configured in ADM, the installed DRAM and DCPMM memory are
exposed to the OS as distinct physical memory devices. Starting with Linux
4.0, both memory types can be abstracted as two distinct non-uniform memory
access (NUMA) nodes. The (virtual) pages of an application can be placed at
either node and are directly accessed through load and store operations.
Therefore, applications have access to a larger aggregate main memory
capacity, since the DRAM capacity is no longer hidden as a cache. In this
setting, the default NUMA placement policy of Linux dictates that once a page
is _first-touched_ it is placed on the fastest node (DRAM) as long as it has
free space; otherwise, the slowest node (DCPMM) is selected. As with any NUMA
system, Linux’s _numactl/libnuma_ interface [21] can be used to implement
different policies.

###  2.3 Related work

###  2.4 Heterogeneous memory hierarchies (HMHs)

Historically, data placement in traditional HMHs (such as DRAM, SSD, disk) has
relied on inclusive caching policies. In such policies, any block from a lower
tier needs to be copied to the top tier to be directly accessed from there.
These are well-studied and a good fit for memory hierarchies whose adjacent
layers differ in cost, capacity and performance by several orders of
magnitude, and where only the fastest layer can be accessed directly by
applications [5].

However, the appearance of new memory technologies has brought in new HMHs
that invalidate these fundamental assumptions and call for radically different
approaches to data placement. Examples include the NUMA architectures that
arise in multi-socket systems [11, 28], high-bandwidth memory+DRAM memory
hierarchies [19], CPU-GPU memory hierarchies [13], or software-defined far
memories [23], among others [61]. In such memory hierarchies, the traditional
caching principle that each requested block needs to be brought into and
served from the fastest memory becomes obsolete. In fact, Zhang et al. [61]
have shown that optimal data placement policies resort to so-called _cache
bypassing_ strategies [33] that intentionally place some blocks in a slower
memory to be accessed directly by applications.

###  2.5 DRAM+PM memory hierarchies

More recently, DRAM+PM memory hierarchies have emerged as an additional case
of a non-traditional HMH. Triggered by the initial promises of the rise of PM
technologies – such as phase-change memory (PCM), Resistive random access
memory (ReRAM), among others [6] –, several works have proposed data placement
techniques for DRAM+PM systems. While, in theory, the main design principles
of data placement in the above-mentioned memory hierarchies also apply to
DRAM+PM systems, PM has distinctive performance and power characteristics,
notably the pronounced read-write access performance asymmetry [39, 56].

To address the problem of data placement in DRAM+PM systems, some authors
proposed programming or compile-time approaches. A first category is
handcrafted applications which, using programming interfaces such as
libvmmalloc library from the Persistent Memory Development Kit (PMDK) [45],
explicitly allocate each object on the most appropriate tier [52, 43, 29].
Alternative schemes resort to profiling-based code analysis to either guide
[12, 53, 40] or even automate [47, 55, 35, 9] object allocation decisions.
Alternatively, runtime systems for specific programming languages have bee
proposed to manage object placement [57].

The scope of our paper is on dynamic approaches to data placement for DRAM+PM.
Dynamic approaches typically manage data placement on a page granularity,
therefore we hereafter designate them as _tiered page placement_ approaches.
In contrast to static compile-time approaches, dynamic page placement
typically incurs no programming burden, easily supports legacy applications,
can dynamically adapt to workload changes by migrating pages across tiers, and
naturally manages multiple concurrent applications from a system-wide
perspective.

Proposed system |  HMH assumptions |  Page placement policy |  Page selection criteria |  Page selection algorithm |  Modifications |  Full-fledged implementation |  Evaluated on DCPMM
---|---|---|---|---|---|---|---
CLOCK-DWF [27] |  DRAM+PCM |  Partitioned |  Hotness+r/w |  CLOCK |  OS |  |
M-CLOCK [26] |  DRAM+PCM |  Fill DRAM first |  Hotness+r/w |  CLOCK |  OS |  |
AC-CLOCK [20] |  DRAM+PCM |  Fill DRAM first |  Hotness+r/w |  CLOCK |  HW+OS |  |
AIMR [48] |  DRAM+PCM/ReRAM |  Fill DRAM first |  Hotness+r/w |  CLOCK+LRU |  HW+OS |  |
CLOCK-HM [8] |  DRAM+PCM |  Fill DRAM first |  Hotness+r/w |  CLOCK+LRU |  HW+OS |  |
Seok et al. [46] |  DRAM+PCM |  Fill DRAM first |  Hotness+r/w |  LRU |  HW+OS |  |
DualStack [62] |  DRAM+PCM |  Fill DRAM first |  Hotness+r/w |  LRU |  HW+OS |  |
HeteroOS [19], Nimble [59] |  MCDRAM+DRAM+NVM |  Fill DRAM first |  Hotness |  LRU |  OS |  ✓ |
UIMigrate [49] |  DRAM+PCM |  Fill DRAM first |  Hotness |  LRU |  HW+OS |  |
TwoLRU [44] |  DRAM+PCM |  Fill DRAM first |  Hotness+r/w |  LRU |  HW+OS |  |
Tiered AutoNUMA [16] |  DRAM+DCPMM |  Fill DRAM first |  Hotness+r/w |  LRU |  OS |  ✓ |  (✓)
Thermostat [1] |  DRAM+3D XPoint |  Fill DRAM first |  Hotness |  TLB misses |  OS |  ✓ |
Memos [30] |  DRAM+NVM |  Fill DRAM first + bandwidth balance |  Hotness |  TLB misses+CLOCK |  OS |  ✓ |
Yu et al. [60] |  DRAM-PCM |  Bandwidth balance |  n/a |  n/a |  |  |
HyPlacer |  DRAM-DCPMM |  Fill DRAM first |  Hotness+r/w |  CLOCK+PCMon [36] |  OS (1 line) |  ✓ |  ✓
TABLE I: Comparison of proposals for tiered page placement on DRAM+PM systems.

Table I summarizes the main proposals for tiered page placement in DRAM+PM
systems and the respective design choices. The majority of these proposals
precedes the commercial availability of PM, with DCPMM. Therefore, they are
implemented and evaluated on simulated or emulated environments, in most cases
calibrated according to performance predictions for PCM PM [54, 24, 42, 25,
7], which is not the technology used in DCPMM. Furthermore, many proposals
simply overlook any integration into a real system stack – they either focus
on the algorithms and dismiss how they can be implemented, or resort to _ad
hoc_ hardware extensions, making it impossible to implement them on off-the-
shelf PM-based systems.

We can categorize three main placement policies among the proposals listed in
Table I. Some adopt a _partitioned_ placement policy, which dynamically
classifies each page as DRAM-bound or PM-bound according to simplistic
criteria that merely depend on each page’s recent access history. Such
classification then dictates at which tier each page resides (with some
exceptions, e.g., if DRAM is full). A common such criterion is to classify
read-dominated pages as PM-bound, motivated by a simplistic assumption that
the read performance of PM is comparable to DRAM. In this policy, pages are
then migrated to DRAM when writes are detected. Proposals such as CLOCK-DWF
[27].

An alternative policy tries to _fill DRAM first_ as pages are accessed and,
once a given DRAM occupancy threshold is reached, selects a set of pages to be
spilled to PM. As Table I shows, this is the most prevalent policy among the
existing proposals. To select which pages to spill, some proposals use
coldness as the single criterion, which conveniently allows them to reuse
existing traditional cache replacement policies, such as LRU or CLOCK. For
instance, HeteroOS [19] and Nimble [59] leverage the active and inactive page
lists that Linux maintains concerning each NUMA node, and exploits them to
implement tiered page placement. The coldness criterion can also be combined
with read and write intensity, motivated by the read/write performance
asymmetry of PM. This combination prioritize read-intensive cold pages over
write-intensive ones as candidates to demote to PM [26, 20, 48, 8, 46, 62, 44,
16].

Finally, a third placement policy is based on the observation that, with a
pure _fill DRAM first_ strategy, bandwidth-intensive workloads might saturate
DRAM bandwidth while not taking advantage of the available PM bandwidth.
Hence, a _bandwidth balance_ strategy tries to distribute hot pages across
DRAM and PM, in some appropriate ratio, with the goal of maximizing the
aggregate bandwidth that applications can attain when accessing different
pages in parallel [60, 30].

At the core of the above placement policies lie some mechanism to track page
accesses, whose output guides the placement decisions. In recent years,
different techniques have been proposed to tracking page accesses in DRAM-PM
page placement systems, ranging from (i) monitoring page table access bits as
in traditional page replacement algorithms (e.g. [19]), (ii) possibly
complemented with TLB miss interception (e.g., [1, 30]), to (iii) hardware
extensions that track additional per-page metrics [20, 48, 8, 46, 62, 49, 44].

Their accuracy vs. runtime overhead trade-offs differs substantially, as
previously authored noted (e.g., [30, 1]). Furthermore, different techniques
require different levels of modifications to the underlying system stack, from
incremental changes to the OS kernel (i) or deeper changes (ii), to modified
hardware (iii). Some recent proposals [1, 30] resort to hierarchical
approaches, which apply different tracking techniques at different levels of
granularity (from contiguous sets of pages to individual pages) to improve
high tracking accuracy at low cost.

Finally, another core component of these solutions is the page migration
mechanism. Although Linux has native support for page migration across DRAM
and PM nodes, some works have proposed optimized migration implementations.
These improvements are achieved by reducing the page management costs
associated with migration [22, 59], exploiting bandwidth-optimized
alternatives to copy page contents across memory nodes [30, 59] (namely,
exploiting multi-threaded or DMA-based data movement), or providing new page
exchange primitives [59].

###  2.6 Tiered page placement with DCPMM

Since the availability of DCPMM, there are some initial studies of data
placement in real DRAM+DCPMM systems. Some authors studied the system
performance of DCPMM [39, 56, 32, 14], and others have manually modified data
placement in either mini or real applications and evaluated the performance
benefits in DRAM+DCPMM systems [37, 52, 43, 29, 51, 58]. Finally, recent
profiling-based proposals to guide or automate static data placement have
already been implemented and evaluated with real DCPMM [9, 38]. Still, with
respect to _dynamic_ tiered page placement, to our knowledge, no previous work
has systematically studied it on real DCPMM-based systems with realistic
applications (the only exception being the tiered Autonuma extension [16],
whose documentation includes a brief evaluation on a synthetic benchmark).
That is the goal of the following sections.

##  3 Insights from real DRAM+DCPMM systems

This section revisits the main design choices of previous proposals for tiered
page placement in DRAM+PM HMAs, in light of our experimental validation of
some of the assumptions that underlie such choices. Using a real machine
equipped with DCPMM (see Section 5 for detailed specifications), we conduct a
set of focused experiments with the purpose of questioning whether each
assumption, and the design choices that depend on it, holds with DCPMM. As we
show next, this empirical evidence provides us with important insights
concerning which design choices are most effective in practice, and which ones
can or should be avoided.

A tiered page placement policy determines, at each instant, what is the set of
pages that resides at each memory tier. In this context, there are several
factors that influence the overall performance and therefore serve as input to
these policies. Most notably, state-of-the-art proposals measure or estimate
per-page hotness and per-page read/write intensity.

We start by studying how these specific factors affect the performance of
different tiered page placement policies. In this initial study, we consider a
data set, which we divide into inactive and active pages. Active pages are
accessed by a multi-threaded application (with as many threads as the
available hardware threads), where threads perform sequential111Due to space
limitations, we omit random-access workloads. We note that switching to
random-access workloads amplifies the per-access costs (when comparing DCPMM
to DRAM) that we present next. accesses to non-overlapping regions. Inactive
pages, in turn, are never accessed during our experiments. For each test, we
vary two main dimensions of the workload: i) the access demand, which
determines the hotness of the active pages, by varying the time each thread
stalls between memory accesses; and ii) the read/write ratio, ranging from
_all reads_ to a 2:1 read-write ratio. We evaluate each workload with two
static placement configurations: i) all active pages in DRAM, ii) all active
pages DCPMM.

We instantiate the above tests using the Intel Memory Latency Checker (MLC)
benchmark [50] on our DCPMM-equipped machine. MLC measures read latency and
bandwidth of the different tested scenarios.

Figure 2: Latency and bandwidth for DRAM and DCPMM, for different read/write
intensities (different lines) and memory access demands (different points in
each line).

Figure 2 presents the results. Each line represents a workload with a given
read/write ratio, and each point in a line denotes an increasing access demand
level. The set of lines corresponding to a given memory tier delimit its
performance lower and upper bounds (concerning latency and bandwidth). We
refer to the interval defined by each set of lines as the DRAM or DCPMM
regions.

This figure therefore allows us to analyze different placement policies, by
interpreting them as follows. Let us assume that, at some instant, a given
application (or system) has a set of pages in DRAM and another set in DCPMM.
Given each set, let us consider the overall access demand and read/write
intensity of the accesses that target the pages in that set (i.e., in the
corresponding memory tier). For each tier, Figure 2 enables us to identify a
point that corresponds to such access demand and read/write intensity in the
associated tier’s region, thus obtaining the corresponding read latency and
bandwidth.

###  3.1 _Partitioned_ policy.

As a first policy that we analyze, consider a _partitioned_ page placement
scheme, which places every read-dominated pages in DCPMM and the remaining
pages in DRAM (as in [27]). In this case, the DCPMM point is located somewhere
in the _all reads_ line of DCPMM, depending on the access demand to such
pages. We observe, in this case, that with workloads that have a small subset
of read-write active pages, a considerable portion of DRAM space will be
unused. Such space would be used more efficiently if it served a fraction of
the active read-only pages, with the associated performance gains. By
comparing the _all reads_ curve of DCPMM with its DRAM counterpart, we see
that, by not placing read-only pages in free DRAM space, the _partitioned_
policy can lead to up to 11.3x latency costs and up to a 2x drop in peak
bandwidth (when such pages are concerned).

Observation 1: for workloads whose set of read-write active pages is smaller
than DRAM, the _partitioned_ policy can incur substantial latency and
bandwidth costs.

###  3.2 _Fill DRAM first_ policy.

In contrast, a _fill DRAM first_ policy is able to mitigate the above
inefficiency. Hence, such policy is more suitable to real DCPMM-based systems.
As discussed in Section 2.3, while some existing proposals exclusively guide
their placement decisions by the page hotness metric, more sophisticated
designs combine hotness with per-page read/write intensity. While more
elaborate, the latter try to exploit the strong read vs. write access
asymmetry of DCPMM.

By analyzing Figure 2, we can infer how much taking read/write intensity into
account matters to tiered page placement. As an example, let us first consider
that, with a policy that is agnostic of read/write intensity, the set of
(hotter) active pages in DRAM has a similar read/write intensity as the set of
(colder) active pages in DCPMM. Further, let us assume that the average
read/write ratio of both sets is 2R:1W. Hence, the access performance to the
active pages in each set is represented by some point in the 2R:1W line of the
corresponding tier in Figure 2 (the actual location of such points depends on
the access demand that the application places on each set of pages). If,
instead, we consider a policy that favors write-intensive pages in DRAM and
read-intensive pages in DCPMM, this change can be visualized as moving the
DRAM point upwards (to a more write-intensive mix of pages) and moving the
DCPMM point downwards (to a more read-dominated mix).

Figure 2 enables us to quantify the potential impact of this approach. While
access demand is low (i.e., the first points of each curve), the different
lines that characterize each tier are relatively overlapping; therefore, for
low access demand, a read/write-aware policy is not expected to yield evident
performance gains. Still, as we increase access demand to higher levels, the
DCPMM curves start diverging substantially after the 20,000 MB/sec limit,
exposing increasingly prominent read vs. write access asymmetries. In
contrast, the DRAM curves only express clear read vs. write asymmetries when
stressed at extreme levels (beyond the 60,000 MB/sec limit), but always with
much smaller discrepancies than those observed with DCPMM. Consequently, for
workloads with medium to high bandwidth intensity, prioritizing write-
intensive pages in DRAM introduces large performance gains when pages in DCPMM
are accessed, while imposing residual to modest performance costs to DRAM
accesses. Hence, the net balance will, in many cases, be positive by a
considerable margin.

Observation 2: For bandwidth-intensive workloads, considering read/write
intensity when choosing which pages to migrate in a _fill DRAM first_ policy
can lead to substantial performance gains when compared only considering per-
page hotness.

Figure 3: Effective bandwidth gains achievable by an ideal _bandwidth balance_
with read-only workloads of varying memory access demand levels, under
different memory module configurations.

###  3.3 _Bandwidth balance_ policy.

As a final question, we wish to quantify what are the potential benefits of a
_bandwidth balance_ policy in contemporary DCPMM-based systems, when compared
to a simpler _fill DRAM first_ approach. A _bandwidth balance_ policy tries to
balance active pages across DRAM and DCPMM such that the application will
distribute its concurrent accesses across both tiers, rather than
concentrating its activity on a single tier. The goal is that the application
benefits from the _aggregate_ bandwidth that both tiers provide, which in
theory is larger than the bandwidth of the DRAM tier only.

A noteworthy observation from the previous insight is that the additional
bandwidth that DCPMM can contribute varies substantially, depending on the
read/write intensity. In fact, for write-intense workloads, DCPMM bandwidth
becomes so low that its contribution to the aggregate bandwidth becomes
residual, as Figure 2 shows. Hence, in order to obtain relevant bandwidth
gains, a _bandwidth balance_ must be able to select only read-dominated active
pages to reside in DCPMM.

Therefore, to quantify the potential advantage of an ideal _bandwidth balance_
, we return to our initial benchmark, and focus exclusively on the _all reads_
workload – i.e., the best-case scenario for _bandwidth balance_. In contrast
to the previous experiment, this time we distribute pages across tiers
according to different ratios (100% in DRAM, 95%, 90%, …), using weighted-
interleaved placement [15]. Also, this time we vary the number of active
threads to test different memory access rates. For each access demand value,
we select the DRAM/DCPMM distribution ratio that obtained the optimal
performance (more precisely, that minimized execution time).

The experiments were performed on our DCPMM-based machine with all available
memory channels populated. We considered different physical memory
configurations, by varying the number of channels used by DRAM and DCPMM
modules, from 3:3 (lower DCPMM bandwidth) to 1:5 (higher DCPMM bandwidth).
Figure 3 presents our findings.

A first finding is that a balanced placement is only advantageous when
bandwidth intensity is high enough. Up to 8 (with 2:4 and 1:5) and 12 threads
(with 3:3), the best performing configuration is to simply place every page in
DRAM. The effective advantage of _bandwidth balance_ only becomes noticeable
at very high access demand rates. To understand this observation, recall from
Figure 2 that DRAM latency is much lower than DCPMM when DRAM bandwidth is not
saturated. Therefore, the break-even point for _bandwidth balance_ only occurs
with bandwidth-intensive workloads that are able to saturate DRAM bandwidth.

Furthermore, even when we consider bandwidth-intensive workloads, the actual
gains of _bandwidth balance_ turn out to be relatively disappointing, yielding
modest speedups (at most 1.13x). These results show that the actual aggregate
bandwidth that _bandwidth balance_ may achieve is, in fact, much lower than
the sum of the nominal peak bandwidth of DRAM and DCPMM.

Observation 3: The potential advantage of a _bandwidth balance_ policy is very
limited in contemporary DCPMM-based systems.

It is worth remarking that, while our machine is populated with one DRAM or
DCPMM module per channel, some motherboards support up to two modules per
channel. However, while at most one DCPMM module can be installed per channel,
a fully populated channel will have an DCPMM module and a DRAM module
competing for the channel’s bandwidth, which further reduces the perceived
DCPMM bandwidth, hence is not expected to change the above results.

##  4 HyPlacer

In this section we present HyPlacer, a system that makes page placement
decisions based on the lessons from the previous section. We start by mapping
those insights to the key design decisions, and then detail the design and
implementation of the system.

###  4.1 From insights to architecture

Guided by the insights we draw in the previous section, we exclude from our
design both the _partitioned_ and the _bandwidth balance_ policies (based on
Observations 1 and 3). Instead, based on Observation 2, we adopt a _fill DRAM
first_ policy that selects pages to migrate based on a combination of per-page
hotness and read/write intensity.

More precisely, HyPlacer collects page and bandwidth metrics at runtime in
order to perform page placement decisions that take advantage of each tier’s
characteristics.To this end, HyPlacer classifies pages into three different
categories: write-intensive, read-intensive and cold. Then, the algorithm
keeps as many write-intensive pages as possible in DRAM. If these do not fully
occupy DRAM, HyPlacer then prefers read-intensive pages over cold pages in the
faster tier. As we will show, this simple set of design choices coupled with a
portable implementation that fit seamlessly into Linux with minimal kernel-
mode components allows HyPlacer to be more efficient and practical than
existing proposals.

###  4.2 System design

HyPlacer manages page placement within a socket with a DRAM and a DCPMM (in
ADM) tier. In multi-socket systems, multiple instances of HyPlacer run
independently, each managing its own socket. For simplicity of presentation,
in the following description we assume a single-socket scenario.

The design of HyPlacer is based on the key idea of having a set of properties
about the page distribution that it tries to maintain, and then react by
migrating pages only in case those target properties are not met. This target
suitability of the current page distribution is defined by the following three
criteria:

  * •

DRAM has enough free space to allow newly referenced pages to fit in the
faster tier. These pages are expected to be accessed frequently after their
allocation, due to the _temporal locality_ principle. Thus, HyPlacer maintains
a defined buffer of free space in DRAM by demoting pages eagerly, before it is
depleted.

  * •

DCPMM’s write throughput is nominal, indicating that the tier does not contain
a significant amount of frequently modified pages.

  * •

If DRAM is at capacity but the DCPMM’s write threshold is substantial, then
this is also considered to be on target, since no pages can be exchanged
between both tiers, such that the threshold is reduced.

If the current distribution does not meet any of these criteria, HyPlacer
devises a new placement decision that corrects it, migrating a subset of pages
in a given orientation.

To select which pages to migrate, HyPlacer leverages the unmodified page table
walk (pagewalk) and PTE bit manipulation mechanisms, implemented in the Linux
kernel. These mechanisms have a relatively stable implementation, which
benefits HyPlacer by making it compatible with a wide range of kernel
versions. Furthermore, configuring HyPlacer on a new kernel version requires
only a single line of code, which exports the pagewalk routine, making it
available to our solution.

In demotion scenarios, we apply concepts from the traditional CLOCK algorithm,
modified to separate intensive pages into read- and write-dominated. Page
promotion, on the other hand, applies a novel delay mechanism, which allows
HyPlacer to identify recently accessed and modified pages in the DCPMM tier
with low overhead.

HyPlacer also implements an exchange-based migration technique, using only
pre-existing system calls, wherein an equal number of pages are switched
between both tiers, thus preserving their current allocation.

###  4.3 Architecture

Figure 4: HyPlacer architecture overview.

Figure 4 provides an overview of HyPlacer, presenting its components and how
they interact. Our solution consists of two main components: Control and the
Page Selection Module (SelMo).

Control is an elevated process running in user-space, which is responsible for
formulating and putting into effect new placement decisions, and
binding/unbinding applications to our solution. In order to formulate new
decisions, the component leverages Processor Counter Monitor (PCMon) [36],
which periodically outputs the current throughput per node to a shared text
file.

To use kernel-implemented mechanisms, HyPlacer integrates a kernel module,
named SelMo. The module selects pages belonging to bound processes, in order
to carry out the Control’s decisions.

The dual-component architecture achieves a small footprint in kernel-space,
since all mechanisms related to devising and carrying out new placement
decisions are offloaded to the user-space component.

###  4.4 Implementation

Overall, HyPlacer leverages existing page management mechanisms already
supported in Linux: (i) existing page walking mechanisms, (ii) the page
table’s dirty and reference bits, managed by the memory management unit (MMU),
(iii) the move_pages syscall to migrate pages between tiers, and (iv)
Processor Counter Monitor (PCMon) [36], which allows HyPlacer to determine the
bandwidth usage of each of the memories with hardware counters available in
most modern Intel CPUs.

Control periodically monitors current memory usage and throughput values per
NUMA node. It is also able to detect the presence of write-intensive pages in
DCPMM, without communicating with SelMo, by reading the information file
generated by PCMon. Depending on the collected metrics, Control devises a new
placement decision and sends a PageFind request to SelMo. The request contains
the number of pages to find and specifies the selection criteria, or mode.

Mode | Tier Scope | Goal
---|---|---
DEMOTE | DRAM |  Demote cold pages
PROMOTE | DCPMM |  Promote pages
PROMOTE_INT | DCPMM |  Promote only intensive pages
SWITCH | Bidirectional |  Switch intensive with cold pages
DCPMM_CLEAR | DCPMM |  Clear the R/D bits from all resident pages
TABLE II:  PageFind modes and goal.

In Table II we summarize the multiple modes, describing the tier from which
they select pages, and their respective goals. By design, DRAM has a defined
maximum usage threshold below its actual size. Above the threshold, HyPlacer
considers that the tier is full or near to depletion, and requests DRAM-
resident cold pages to demote from SelMo, via a DEMOTE PageFind.

Similarly, DCPMM has a defined write throughput threshold. If DCPMM’s current
throughput is above the defined threshold, Control requests intensive pages
and promotes them to DRAM. If DRAM is above its usage threshold, an equal
number of pages must be demoted, such that the free space buffer is preserved.
Therefore, a SWITCH PageFind is sent to the module. Otherwise, it tries to
maximize the faster tier’s utilization, by promoting as many intensive pages
as possible such that the usage threshold is not surpassed, by requesting a
PROMOTE_INT PageFind.

Inversely, if the DCPMM’s throughput is below its threshold, two decisions can
be performed. If DRAM has enough available space, Control allows cold pages to
be eagerly promoted, via a PROMOTE PageFind. Otherwise, if DRAM is near
depletion, a DEMOTE PageFind is requested, such that cold pages are demoted.

Before sending any PageFind request that will promote pages, Control requests
SelMo to clear the R/D bits of all PTEs pointing to pages in the DCPMM tier,
via DCPMM_CLEAR PageFind, after which it waits for a configurable delay. Delay
affects the access frequency at which a page is considered intensive. Pages
that are accessed or modified during the delay interval are considered read-
or write-intensive, while all others are classified as cold.

After receiving a PageFind request, SelMo iterates over each bound process’
page table, in order to select which pages Control should migrate. The module
leverages the kernel-implemented routine walk_page_range() , which iterates
over a defined virtual address range. Although this routine is not originally
invocable from arbitrary kernel modules, we expose it to kernel modules by
changing a single line of code in the Linux kernel. This is the only change to
kernel code that HyPlacer requires.

SelMo passes a PTE callback as an argument to the routine, so that it can
observe and manipulate each PTE’s R/D bits. Since we want to perform different
operations depending on the goal of the PageFind request, we define multiple
callbacks, one for each mode.

If the callback’s goal is to demote pages, then it clears the R/D bits of all
pages that are not selected to demote. If one such page is referenced
thereafter, the memory management unit (MMU) sets its PTE’s reference bit; and
also its modified bit, in the event of a store operation. In contrast, if the
page is not accessed until the next page table iteration, then it is suitable
for demotion.

If, otherwise, the callback’s goal is to promote pages, then PTEs are expected
to have both their R/D bits unset, since they have been recently cleared by
the module. In this scenario, the MMU may change the PTE’s R/D bits, so that
the respective page is suitable for promotion over a next page table
iteration. Therefore, promotion callbacks do not directly manipulate R/D bits.
Instead, the algorithm deems a page in the DCPMM intensive if referenced
during the delay window, and write-dominated if modified.

When: (i) the number of selected pages exceeds the required amount set in the
request; or (ii) the process has iterated over all PTEs, the page selection
phase ends. At this point, the last PTE’s address and PID are stored and the
page selection phase ends. For each tier, the module keeps two last address
and PID pairs, which set the start of the next page selection phase for that
tier. Thus, PTEs that have not been inspected for longer are prioritized for
migration over recently seen ones. Then, a reply-back phase begins, where
SelMo prepares a final page array to be sent back to Control, containing all
selected pages, after which the latter migrates them.

##  5 Evaluation

In this section we evaluate the performance of HyPlacer when managing
different realistic workloads and compare it against: (i) HMH-aware dynamic
placement solutions recently proposed in literature, and (ii) placement
options that are currently available in off-the-shelf DCPMM-equipped Linux
systems. In particular, our evaluation studies the throughput and energy
consumption of each placement alternative with workloads of different sizes,
read/write ratios, locality, access patterns and memory intensities.

Section 5.1 starts by detailing the experimental setup. Section 5.2 then
presents the throughput and power results, discussing how the different
systems perform when managing realistic applications with large data sets.
Finally, Section 5.3 analyzes the overheads of the different solutions when
used in a worst-case scenario of small data sets.

###  5.1 Experimental setup

Machine. We use a dual-socket machine with Intel® Xeon® Gold 5218 CPU, running
at 2.30GHz. Each socket is populated with 2 DRAM modules (2x16GB DDR4 2666
MT/s) and 2 DCPMM modules (2x128 GB, DDR-T 2666MT/s Series 100), resulting in
a total of of 32GB of DRAM and 256GB of DCPMM per socket. Each module is
installed at a distinct memory channel.

Unless otherwise noted, all experiments run on a Debian-based v5.8.5 kernel.
We set the swappiness value to 0, which effectively disables swapping in our
experiments. We restrict every experiment to a single socket, by using
_numactl_ to bind each application to exclusively execute on the cores of the
first socket and place its pages in the local DRAM and DCPMM nodes. Finally,
we disable transparent huge pages (THP), since we observed that the current
native support in Linux for THP migration is very inefficient (since each THP
is split and migrated as 4KB pages); the only exception is explained below.

Evaluated solutions. We evaluate a wide set of alternative approaches to
tiered page placement.

On a first category, we consider the two available alternatives that are
available by default in current Linux-based systems with DCPMM. Namely, we
consider: (a) DRAM and DCPMM configured in ADM, with the default first-touch
NUMA policy, without any dynamic placement solution applied (ADM-default); and
(b) DCPMM configured in MemM. As explained in Section 2, the former is a two-
tiered configuration with no tier migration, whereas the latter provides a
hardware-managed caching algorithm, which dynamically caches and evicts
intensive data to and from DRAM.

On a second category, we evaluate a selection of recent proposals for tiered
page placement from the literature. As a selection criteria, we considered all
the proposals in Table I that were designed and implemented to run on
commodity systems. This comprises:

  1. 1.

autonuma: Intel’s extension of the original AutoNUMA NUMA scheduling mechanism
of Linux [16], which adds tiered page placement to AutoNUMA’s original
features. Since we confine our experiments to a single-socket, autonuma does
not place or migrate threads or pages across sockets, which is its most common
functionality. Instead, it simply manages tiered page placement between DRAM
and DCPMM on a single socket.

We used the tiering-0.4 version [17], which was the most up-to-date documented
version at the time of testing, based on the v5.5 Linux kernel. We configure
the kernel and run the post boot setup as proposed in the documentation, using
the recommended settings for performance experiments.

  2. 2.

memos, the tiered page placement of the Memos [30] OS kernel, an extension of
the Linux kernel. Memos employs an intricate adaptive and read/write-aware
_bandwidth balance_ policy. Since Memos’ code was not publicly available, we
implemented the proposed page placement policy on top of the architecture of
HyPlacer, relying on the mechanisms implemented in HyPlacer to monitor page
hotness and read/write intensity. Note that we omit features proposed in [30]
such bank imbalance, alternative migration techniques, and an in-house TLB
miss profiler, as implementing these required deep kernel modifications.

Memos’ original parametrization [30] is suited to low footprint workloads,
only migrating a maximum of 10,000 pages at each cycle, which correlates to
1MB/s. Therefore, we change it to make it as competitive as possible with our
chosen workloads, as some will be multiple times larger than DRAM size
(¿27GB). Firstly, we tighten memos’ periodicity from 40s to 4s and, in order
to fit in the new 4s period, lower the number of required page classifications
to a single one, sacrificing accuracy for performance. Secondly, we increase
the maximum number of pages that can be migrated in a given period to 10x its
original value, allowing 100,000 pages to be promoted. Both changes increase
memos’ migration rate-limit a hundredfold, to 100MB/s.

  3. 3.

nimble, the tiered page placement of Nimble [59]. This is an incremental
extension to the active and inactive page lists maintained by Linux for each
NUMA memory node, which distinguish hot and cold pages, respectively. Using
the same strategy originally proposed in HeteroOS [19], Nimble uses such lists
to implement a _fill DRAM first_ policy. We directly used the open source
Nimble patch to the Linux v5.6rc6 kernel, with the default parameters
described in the paper. Since Nimble also incorporates an optimized support
for THP migration, we enabled it for the experiments with this system, in
order to take advantage of its full feature set.

Finally, a full-fledged implementation of HyPlacer was tested and compared
against the above alternatives. We used the following main parameters: a DRAM
occupancy threshold of 95%; at most 128K pages migrated at each activation; an
DCPMM bandwidth threshold of 10 MB/sec; and an access bit clearance delay of
50 ms.

Workloads. We evaluate multiple applications from the NAS Parallel Benchmark
(NPB) [2, 34] suite, extracted from the OpenMP [10] version of NPB v3.4.1. NPB
provides benchmarks that mimic common access patterns in computational fluid
dynamics applications, and was designed to evaluate the performance of
parallel supercomputers. Within the NPB suite, the selection criteria was to
include the applications that could be instantiated with larger data sets
(much larger than the available DRAM, 32 GB). As a result, we used the BT, FT,
MG and CG applications. Except for the parameters affecting data set size, we
used the default arguments. All applications run with as many hardware threads
as possible in the CPU, 32 threads.

Table III summarizes the selected applications. Overall, the applications are
characterized by different memory intensities, as well as read vs. write
ratios.

Benchmark |  Read/write ratio |  Data Sets Sizes (GB) |  Average execution time
---|---|---|---
BT |  3.5R:1W |  28.4 (S), 39.1 (M), 53.9 (L) |  9m (S), 16m (M), 45m (L)
FT |  1.7R:1W |  20 (S), 40 (M), 80 (L) |  3m (S), 1h38m (M), 6h29m (L)
MG |  4R:1W |  26.5 (S), 74.3 (M), 131 (L) |  6m (S), 1h16m (M), 2h53m (L)
CG |  $>$60R:1W |  18 (S), 39.8 (M), 150 (L) |  12m (S), 41m (M), 3h58m (L)
TABLE III:  Summary of evaluated applications.  Figure 5: Throughput speedup
compared to ADM-Default.

Each application is tested with three data set sizes: small, which totally
fits in DRAM; medium, which is approximately 1.5x DRAM size; and large,
approximately 3.5x DRAM size. The small size is a worst-case scenario for
which tiered page placement brings no effective advantage to a trivial static
placement in DRAM. The medium and large sizes are the most relevant ones, as
they require the supplementary memory capacity that DCPMM provides, hence it
is in these scenarios that tiered page placement solutions are expected to
benefit the application.

For each experiment, the results we show next are averaged over 3 runs.

Figure 6: Per-access memory energy consumption compared to ADM-Default.

###  5.2 Performance results with large data sets

Figure 5 presents a set of plots showing the performance speedup for each of
the BT, FT, MG, and CG benchmarks under both medium and large size data sets
compared to the ADM-default configuration, as well as the geometric average
speedup of all four benchmarks.

We observe that nimble performs significantly worse than the other solutions
for most of the cases, being at par or worse relative to the ADM-default
configuration. Among other factors, this can be explained by _nimble_ ’s
simplistic page selection mechanism, which exclusively considers page hotness,
hence is not aware of the read/write intensity of the pages it decides to
migrate across tiers. Furthermore, this also reflects a sub-optimal parameter
choice, which was originally defined based on innacurate assumptions about the
real persistent memory.

The memos solution also presents lower speedups than the remaining tested
solutions in the majority of scenarios, with an average 28% performance
reduction compared to the ADM-default configuration. This solution performs
best in the very read-intensive CG workloads, but even for this case it is
only better than HyPlacer for the medium-sized load. The reasons for this low
performance are the poor initial memory placement policy of Memos, which
allocates new pages in DCPMM, and also due to its ineffective bandwidth-aware
promotion mechanism, which often fails to saturate DRAM throughput.

Comparing the other solutions in the medium data sets, MemM, autonuma, and
HyPlacer show a speedup of 2.5x, 2.3x, and 3.7x on average vs. ADM-default.
The speedup increases in the large parametrizations, to 3.8x, 2.8x, and 5.4x,
respectively.

We observe that HyPlacer’s placement mechanisms improve throughput more than
the autonuma, despite only changing a single line of code in the kernel, and
processing most of the placement decisions in a user-level process. In all
workloads, the ambix configuration grants an average speedup of 4.6x, compared
to autonuma’s 2.6x. Its benefit is most noticeable in the BT workloads, where
autonuma fails to improve the ADM-default configuration, while ambix has an
average 2.25x speedup. In the read-intensive CG workload, autonuma grants
better performance than ambix in the medium parametrization, but falls off in
the large one, with a 4x vs. 11x speedup compared to ADM-default.

MemM performs better than autonuma in the majority of medium and large
workloads. However, ambix surpasses it, having a 45% higher average speedup in
all workloads.

Besides throughput, we measured the energy consumption of each application
with each page placement approach. To measure energy consumption, we used
Linux’s _perf_ tool to sample the energy consumption of the memory system;
more precisely, using _perf stat_ to collect measurements from event
_power/energy-ram_ every second.

Figure 6 depicts the gains in energy for each solution for the same instances
as Figure 5, again using the energy consumption of the ADM-default
configuration as baseline. Note that in Figure 6 higher is better as the
values represent how many times lower the energy consumption is relative to
the baseline.

These results demonstrate that a good page placement decision can have a
significant impact on the energy efficiency of a program. Furthermore,
comparing these two figures, we can observe that the trends of energy gains
are mostly consistent with the throughput speedup values (in Figure 5).

Figure 7: Throughput speedup compared to ADM-Default, with data sets that fit
entirely in DRAM.

###  5.3 Overheads in the worst-case scenario (small data sets)

We now turn our attention to the overheads of the different solutions. To
evaluate that, we consider a worst-case scenario where the data sets of every
application are small enough to fit entirely in DRAM. Of course, in this
scenario, the optimal solution consists in simply placing every page in DRAM,
statically. For this reason, the slowdowns presented in Figure fig:NPB-small
over the baseline are to be expected, representing the overheads associated
with each proposed solution. We observe that in most cases the results are
close to the baseline. Still, ambix presents a higher penalty for the MG and
FT cases, justified by preemptive, unnecessary page migration.

##  6 Conclusions

In this paper, we are the first, to our knowledge, to systematically analyze
tiered page placement on real DCPMM-based systems. Based on an initial
experimental study using a real DCPMM-based system, we reveal that some of the
key design choices in the literature rely on important assumptions that are
not verified in contemporary DRAM-DCPMM memory architectures.

Based on these lessons, we propose HyPlacer, a practical tool for tiered page
placement in _off-the-shelf_ Linux-based systems equipped with DRAM+DCPMM. In
contrast to previous proposals, HyPlacer follows an approach guided by two
main practicality principles: 1) it is tailored to the actual performance
characteristics of off-the-shelf DRAM+DCPMM systems; and 2) it can be
seamlessly integrated into Linux with minimal kernel-mode components, while
ensuring extensibility to other HMAs and other data placement policies.

We evaluate HyPlacer, as well as relevant page placement alternatives, using
realistic benchmarks. We show that HyPlacer outperforms both solutions
proposed in past literature as well as placement options that are currently
available in off-the-shelf DCPMM-equipped Linux systems, with an average
speedup of 4.6x in large footprint workloads, reaching a peak improvement of
11x when compared to the default memory policy in Linux.

## Acknowledgments

This work was partially supported by FCT (UIDB/50021/2020), and EU’s H2020 R&I
programme (EPEEC project, GA 801051).

## References

  * [1] N. Agarwal and T. F. Wenisch, “Thermostat: Application-transparent page management for two-tiered main memory,” in _Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2017, pp. 631–644.
  * [2] D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber _et al._ , “The nas parallel benchmarks summary and preliminary results,” in _Supercomputing’91: Proceedings of the 1991 ACM/IEEE conference on Supercomputing_. IEEE, 1991, pp. 158–165.
  * [3] A. Baldassin, J. a. Barreto, D. Castro, and P. Romano, “Persistent memory: A survey of programming support and implementations,” _ACM Comput. Surv._ , vol. 54, no. 7, Jul. 2021. [Online]. Available: https://doi.org/10.1145/3465402
  * [4] S. Beamer, K. Asanović, and D. Patterson, “The gap benchmark suite,” _arXiv preprint arXiv:1508.03619_ , 2015.
  * [5] L. A. Belady, “A study of replacement algorithms for a virtual-storage computer,” _IBM Syst. J._ , vol. 5, no. 2, p. 78–101, Jun. 1966. [Online]. Available: https://doi.org/10.1147/sj.52.0078
  * [6] J. Boukhobza, S. Rubini, R. Chen, and Z. Shao, “Emerging nvm: A survey on architectural integration and research challenges,” _ACM Trans. Des. Autom. Electron. Syst._ , vol. 23, no. 2, Nov. 2017. [Online]. Available: https://doi.org/10.1145/3131848
  * [7] G. W. Burr, M. J. Breitwisch, M. Franceschini, D. Garetto, K. Gopalakrishnan, B. Jackson, B. Kurdi, C. Lam, L. A. Lastras, A. Padilla _et al._ , “Phase change memory technology,” _Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena_, vol. 28, no. 2, pp. 223–262, 2010.
  * [8] X. Cai, L. Ju, M. Zhao, Z. Sun, and Z. Jia, “A novel page caching policy for pcm and dram of hybrid memory architecture,” in _2016 13th International Conference on Embedded Software and Systems (ICESS)_. IEEE, 2016, pp. 67–73.
  * [9] Y. Chen, I. B. Peng, Z. Peng, X. Liu, and B. Ren, “Atmem: Adaptive data placement in graph applications on heterogeneous memories,” in _Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization_ , ser. CGO 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 293–304. [Online]. Available: https://doi.org/10.1145/3368826.3377922
  * [10] L. Dagum and R. Menon, “Openmp: an industry standard api for shared-memory programming,” _IEEE computational science and engineering_ , vol. 5, no. 1, pp. 46–55, 1998.
  * [11] M. Dashti, A. Fedorova, J. Funston, F. Gaud, R. Lachaize, B. Lepers, V. Quema, and M. Roth, “Traffic management: a holistic approach to memory placement on numa systems,” _ACM SIGPLAN Notices_ , vol. 48, no. 4, pp. 381–394, 2013\.
  * [12] S. R. Dulloor, A. Roy, Z. Zhao, N. Sundaram, N. Satish, R. Sankaran, J. Jackson, and K. Schwan, “Data tiering in heterogeneous memory systems,” in _Proceedings of the Eleventh European Conference on Computer Systems_ , 2016, pp. 1–16.
  * [13] D. Ganguly, Z. Zhang, J. Yang, and R. Melhem, “Interplay between hardware prefetcher and page eviction policy in cpu-gpu unified virtual memory,” in _Proceedings of the 46th International Symposium on Computer Architecture_ , ser. ISCA ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 224–235. [Online]. Available: https://doi.org/10.1145/3307650.3322224
  * [14] S. Gugnani, A. Kashyap, and X. Lu, “Understanding the idiosyncrasies of real persistent memory,” _Proc. VLDB Endow._ , vol. 14, no. 4, p. 626–639, Dec. 2020. [Online]. Available: https://doi.org/10.14778/3436905.3436921
  * [15] D. Gureya, J. Neto, R. Karimi, J. Barreto, P. Bhatotia, V. Quema, R. Rodrigues, P. Romano, and V. Vlassov, “Bandwidth-aware page placement in numa,” in _2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_. Los Alamitos, CA, USA: IEEE Computer Society, may 2020, pp. 546–556. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/IPDPS47924.2020.00063
  * [16] Y. Huang, “autonuma: Optimize memory placement in memory tiering system,” https://lwn.net/Articles/803663/, 2020, [Online; accessed 19-November-2020].
  * [17] Y. Huang and V. Verma, “AutoNUMA: tiering-0.4,” https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git?h=tiering-0.4, 2020, [Online; accessed 19-November-2020].
  * [18] Intel, “Intel® OptaneTM DC Persistent Memory – Quick Start Guide,” https://www.intel.com/content/dam/support/us/en/documents/memory-and-storage/data-center-persistent-mem/Intel-Optane-DC-Persistent-Memory-Quick-Start-Guide.pdf, 2020\.
  * [19] S. Kannan, A. Gavrilovska, V. Gupta, and K. Schwan, “Heteroos: Os design for heterogeneous memory management in datacenter,” in _Proceedings of the 44th Annual International Symposium on Computer Architecture_ , 2017, pp. 521–534.
  * [20] S. Kim, S.-H. Hwang, and J. W. Kwak, “Adaptive-classification clock: Page replacement policy based on read/write access pattern for hybrid dram and pcm main memory,” _Microprocessors and Microsystems_ , vol. 57, pp. 65–75, 2018\.
  * [21] A. Kleen, “A numa api for linux,” _Novel Inc_ , 2005.
  * [22] M. K. Kumar, S. Maass, S. Kashyap, J. Veselý, Z. Yan, T. Kim, A. Bhattacharjee, and T. Krishna, “Latr: Lazy translation coherence,” _SIGPLAN Not._ , vol. 53, no. 2, p. 651–664, Mar. 2018. [Online]. Available: https://doi.org/10.1145/3296957.3173198
  * [23] A. Lagar-Cavilla, J. Ahn, S. Souhlal, N. Agarwal, R. Burny, S. Butt, J. Chang, A. Chaugule, N. Deng, J. Shahid, G. Thelen, K. A. Yurtsever, Y. Zhao, and P. Ranganathan, “Software-defined far memory in warehouse-scale computers,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_ , ser. ASPLOS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 317–330. [Online]. Available: https://doi.org/10.1145/3297858.3304053
  * [24] S. Lai, “Current status of the phase change memory and its future,” in _IEEE International Electron Devices Meeting 2003_. IEEE, 2003, pp. 10–1.
  * [25] B. C. Lee, E. Ipek, O. Mutlu, and D. Burger, “Architecting phase change memory as a scalable dram alternative,” _ACM SIGARCH Computer Architecture News_ , vol. 37, no. 3, pp. 2–13, 2009.
  * [26] M. Lee, D. H. Kang, J. Kim, and Y. I. Eom, “M-clock: Migration-optimized page replacement algorithm for hybrid dram and pcm memory architecture,” in _Proceedings of the 30th Annual ACM Symposium on Applied Computing_ , ser. SAC ’15. New York, NY, USA: ACM, 2015, pp. 2001–2006. [Online]. Available: http://doi.acm.org/10.1145/2695664.2695675
  * [27] S. Lee, H. Bahn, and S. H. Noh, “Clock-dwf: A write-history-aware page replacement algorithm for hybrid pcm and dram memory architectures,” _IEEE Transactions on Computers_ , vol. 63, no. 9, pp. 2187–2200, Sep. 2014\.
  * [28] B. Lepers, V. Quéma, and A. Fedorova, “Thread and memory placement on $\\{$NUMA$\\}$ systems: Asymmetry matters,” in _2015 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 15)_, 2015, pp. 277–289.
  * [29] J. Liu, D. Li, R. Gioiosa, and J. Li, “Athena: High-performance sparse tensor contraction sequence on heterogeneous memory,” in _Proceedings of the ACM International Conference on Supercomputing_ , ser. ICS ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 190–202. [Online]. Available: https://doi.org/10.1145/3447818.3460355
  * [30] L. Liu, S. Yang, L. Peng, and X. Li, “Hierarchical hybrid memory management in os for tiered memory systems,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 30, no. 10, pp. 2223–2236, 2019.
  * [31] H. Luo, T. Shahroodi, H. Hassan, M. Patel, A. G. Yağlıkçı, L. Orosa, J. Park, and O. Mutlu, “Clr-dram: A low-cost dram architecture enabling dynamic capacity-latency trade-off,” in _Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture_ , ser. ISCA ’20. IEEE Press, 2020, p. 666–679. [Online]. Available: https://doi.org/10.1109/ISCA45697.2020.00061
  * [32] V. Mironov, I. Chernykh, I. Kulikov, A. Moskovsky, E. Epifanovsky, and A. Kudryavtsev, “Performance evaluation of the intel optane dc memory with scientific benchmarks,” in _2019 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC)_. IEEE, 2019, pp. 1–6.
  * [33] S. Mittal, “A survey of cache bypassing techniques,” _Journal of Low Power Electronics and Applications_ , vol. 6, no. 2, 2016. [Online]. Available: https://www.mdpi.com/2079-9268/6/2/5
  * [34] NASA, “NAS Parallel Benchmarks,” https://www.nas.nasa.gov/publications/npb.html, 2021, [Online; accessed 20-May-2021].
  * [35] M. B. Olson, T. Zhou, M. R. Jantz, K. A. Doshi, M. G. Lopez, and O. Hernandez, “Membrain: Automated application guidance for hybrid memory systems,” in _2018 IEEE International Conference on Networking, Architecture and Storage (NAS)_. IEEE, 2018, pp. 1–10.
  * [36] OPCM, “Processor Counter Monitor,” https://github.com/opcm/pcm, 2016, [Online; accessed 25-November-2020].
  * [37] O. Patil, L. Ionkov, J. Lee, F. Mueller, and M. Lang, “Performance characterization of a dram-nvm hybrid memory architecture for hpc applications using intel optane dc persistent memory modules,” in _Proceedings of the International Symposium on Memory Systems_ , ser. MEMSYS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 288–303. [Online]. Available: https://doi.org/10.1145/3357526.3357541
  * [38] I. Peng, K. Wu, J. Ren, D. Li, and M. Gokhale, “Demystifying the performance of hpc scientific applications on nvm-based memory systems,” in _2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_ , 2020, pp. 916–925.
  * [39] I. B. Peng, M. B. Gokhale, and E. W. Green, “System evaluation of the intel optane byte-addressable nvm,” in _Proceedings of the International Symposium on Memory Systems_ , 2019, pp. 304–315.
  * [40] I. B. Peng and J. S. Vetter, “Siena: Exploring the design space of heterogeneous memory systems,” in _SC18: International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2018, pp. 427–440.
  * [41] G. O. Puglia, A. F. Zorzo, C. A. F. De Rose, T. Perez, and D. Milojicic, “Non-volatile memory file systems: A survey,” _IEEE Access_ , vol. 7, pp. 25 836–25 871, 2019.
  * [42] M. K. Qureshi, V. Srinivasan, and J. A. Rivers, “Scalable high performance main memory system using phase-change memory technology,” in _ACM SIGARCH Computer Architecture News_ , vol. 37. ACM, 2009, pp. 24–33.
  * [43] J. Ren, J. Luo, I. Peng, K. Wu, and D. Li, “Optimizing large-scale plasma simulations on persistent memory-based heterogeneous memory with effective data placement across memory hierarchy,” in _Proceedings of the ACM International Conference on Supercomputing_ , ser. ICS ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 203–214. [Online]. Available: https://doi.org/10.1145/3447818.3460356
  * [44] R. Salkhordeh and H. Asadi, “An operating system level data migration scheme in hybrid dram-nvm memory architecture,” in _2016 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2016, pp. 936–941.
  * [45] S. Scargall, _Programming Persistent Memory - A Comprehensive Guide for Developers_ , 1st ed. Apress, 2020.
  * [46] H. Seok, Y. Park, K.-W. Park, and K. H. Park, “Efficient page caching algorithm with prediction and migration for a hybrid main memory,” _SIGAPP Appl. Comput. Rev._ , vol. 11, no. 4, pp. 38–48, Dec. 2011. [Online]. Available: http://doi.acm.org/10.1145/2107756.2107760
  * [47] H. Servat, A. J. Peña, G. Llort, E. Mercadal, H. Hoppe, and J. Labarta, “Automating the application data placement in hybrid memory systems,” in _2017 IEEE International Conference on Cluster Computing (CLUSTER)_ , Sep. 2017, pp. 126–136.
  * [48] Z. Sun, Z. Jia, X. Cai, Z. Zhang, and L. Ju, “Aimr: An adaptive page management policy for hybrid memory architecture with nvm and dram,” in _2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems_. IEEE, 2015, pp. 284–289.
  * [49] Y. Tan, B. Wang, Z. Yan, Q. Deng, X. Chen, and D. Liu, “Uimigrate: Adaptive data migration for hybrid non-volatile memory systems,” in _2019 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2019, pp. 860–865.
  * [50] V. Viswanathan, K. Kumar, T. Willhalm, P. Lu, B. Filipiak, and S. Sakthivelu, “Intel memory latency checker,” _Intel Corporation_ , 2013.
  * [51] D. Waddington, M. Kunitomi, C. Dickey, S. Rao, A. Abboud, and J. Tran, “Evaluation of intel 3d-xpoint nvdimm technology for memory-intensive genomic workloads,” in _Proceedings of the International Symposium on Memory Systems_ , ser. MEMSYS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 277–287. [Online]. Available: https://doi.org/10.1145/3357526.3357528
  * [52] M. Weiland, H. Brunst, T. Quintino, N. Johnson, O. Iffrig, S. Smart, C. Herold, A. Bonanni, A. Jackson, and M. Parsons, “An early evaluation of intel’s optane dc persistent memory module and its impact on high-performance scientific applications,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2019, pp. 1–19.
  * [53] S. Wen, L. Cherkasova, F. X. Lin, and X. Liu, “Profdp: A lightweight profiler to guide data placement in heterogeneous memory systems,” in _Proceedings of the 2018 International Conference on Supercomputing_ , ser. ICS ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 263–273. [Online]. Available: https://doi.org/10.1145/3205289.3205320
  * [54] H.-S. P. Wong, S. Raoux, S. Kim, J. Liang, J. P. Reifenberg, B. Rajendran, M. Asheghi, and K. E. Goodson, “Phase change memory,” _Proceedings of the IEEE_ , vol. 98, no. 12, pp. 2201–2227, 2010.
  * [55] K. Wu, Y. Huang, and D. Li, “Unimem: Runtime data managementon non-volatile memory-based heterogeneous main memory,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2017, pp. 1–14.
  * [56] K. Wu, F. Ober, S. Hamlin, and D. Li, “Early evaluation of intel optane non-volatile memory with hpc i/o workloads,” _arXiv preprint arXiv:1708.02199_ , 2017.
  * [57] K. Wu, J. Ren, and D. Li, “Runtime data management on non-volatile memory-based heterogeneous memory for task-parallel programs,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis_ , ser. SC ’18. IEEE Press, 2018.
  * [58] Z. Xie, W. Dong, J. Liu, I. B. Peng, Y. Ma, and D. Li, “MD-HM: memoization-based molecular dynamics simulations on big memory system,” in _ICS ’21: 2021 International Conference on Supercomputing, Virtual Event, USA, June 14-17, 2021_ , H. Zhou, J. Moreira, F. Mueller, and Y. Etsion, Eds. ACM, 2021, pp. 215–226. [Online]. Available: https://doi.org/10.1145/3447818.3460365
  * [59] Z. Yan, D. Lustig, D. Nellans, and A. Bhattacharjee, “Nimble page management for tiered memory systems,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2019, pp. 331–345.
  * [60] S. Yu, S. Park, and W. Baek, “Design and implementation of bandwidth-aware memory placement and migration policies for heterogeneous memory systems,” in _Proceedings of the International Conference on Supercomputing_ , 2017, pp. 1–10.
  * [61] L. Zhang, R. Karimi, I. Ahmad, and Y. Vigfusson, “Optimal data placement for heterogeneous cache, memory, and storage systems,” _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ , vol. 4, no. 1, pp. 1–27, 2020.
  * [62] Z. Zhang, Y. Fu, and G. Hu, “Dualstack: A high efficient dynamic page scheduling scheme in hybrid main memory,” in _2017 International Conference on Networking, Architecture, and Storage (NAS)_. IEEE, 2017, pp. 1–6.