IEEEexample:BSTcontrol

# Exploiting Inter- and Intra-Memory Asymmetries for Data Mapping in Hybrid
Tiered-Memories

Shihao Song<EMAIL_ADDRESS>Drexel University3101 Market
StreetPhiladelphiaPAUSA ,  Anup Das<EMAIL_ADDRESS>Drexel University3101
Market StreetPhiladelphiaPA19104USA and  Nagarajan Kandasamy<EMAIL_ADDRESS>Drexel University3101 Market StreetPhiladelphiaPA19104USA

###### Abstract.

Modern computing systems are embracing hybrid memory comprising of DRAM and
non-volatile memory (NVM) to combine the best properties of both memory
technologies, achieving low latency, high reliability, and high density. A
prominent characteristic of DRAM-NVM hybrid memory is that it has NVM access
latency much higher than DRAM access latency. We call this _inter-memory
asymmetry_. We observe that parasitic components on a long bitline are a major
source of high latency in both DRAM and NVM, and a significant factor
contributing to high-voltage operations in NVM, which impact their
reliability. We propose an architectural change, where each long bitline in
DRAM and NVM is split into two segments by an isolation transistor. One
segment can be accessed with lower latency and operating voltage than the
other. By introducing tiers, we enable non-uniform accesses within each memory
type (which we call intra-memory asymmetry), leading to performance and
reliability trade-offs in DRAM-NVM hybrid memory.

We show that our hybrid tiered-memory architecture has a tremendous potential
to improve performance and reliability, if exploited by an efficient page
management policy at the operating system (OS). Modern OSes are already aware
of inter-memory asymmetry. They migrate pages between the two memory types
during program execution, starting from an initial allocation of the page to a
randomly-selected free physical address in the memory. We extend existing OS
awareness in three ways. First, we exploit both inter- and intra-memory
asymmetries to allocate and migrate memory pages between the tiers in DRAM and
NVM. Second, we improve the OS’s page allocation decisions by predicting the
access intensity of a newly-referenced memory page in a program and placing it
to a matching tier during its initial allocation. This minimizes page
migrations during program execution, lowering the performance overhead. Third,
we propose a solution to migrate pages between the tiers of the same memory
without transferring data over the memory channel, minimizing channel
occupancy and improving performance. Our overall approach, which we call
MNEME, to enable and exploit asymmetries in DRAM-NVM hybrid tiered memory
improves both performance and reliability for both single-core and multi-
programmed workloads.

phase change memory (PCM), DRAM, tiered memory, bitline parasitic, hybrid
memory, non volatile memory (NVM), NBTI, endurance

††conference: Proceedings of the 2020 ACM SIGPLAN International Symposium on
Memory Management; June 16, 2020; London, UK††booktitle: Proceedings of the
2020 ACM SIGPLAN International Symposium on Memory Management (ISMM ’20), June
16, 2020, London, UK††ccs: Computer systems organization Processors and memory
architectures††ccs: Hardware Memory and dense storage††ccs: Hardware Aging of
circuits and systems††ccs: Software and its engineering Main memory††ccs:
Software and its engineering Virtual memory

##  1\. Introduction

DRAM has long been the choice substrate for architecting main memory
subsystems due to its low cost per bit. However, DRAM is a fundamental
performance and energy bottleneck in almost all computer systems (Wulf and
McKee, 1995; Wilkes, 2001; Mutlu and Subramanian, 2015; Mutlu, 2013), and is
experiencing significant technology scaling challenges (Mutlu, 2013; Kang et
al., 2014; Mutlu, 2017; Mutlu and Kim, 2019). DRAM-compatible emerging non-
volatile memory (NVM) technologies such as Flash (Cai et al., 2017), oxide-
based RAM (OxRAM) (Mallik et al., 2017), phase-change memory (PCM) (Wong et
al., 2010), and spin transfer torque magnetic RAM (STT-MRAM) (Apalkov et al.,
2013) can address some of these challenges (Lee et al., 2009a; Kültürsay et
al., 2013; Song et al., 2019, 2020; Ren et al., 2015; Lee et al., 2010a;
Qureshi et al., 2009). However, they are usually slower than DRAM and have
limited endurance.111NVM’s endurance ranges from $10^{5}$ writes for Flash to
$10^{10}$ writes for OxRAM, and PCM in between, with $10^{7}$ writes. Modern
computing systems are therefore embracing hybrid memory designs comprising of
DRAM and NVM (Arafa and Ramanujan, 2018). These systems combine the best
properties of both memory technologies to improve latency, reliability,
capacity, and cost. The non-volatile 3D XPoint memory (Bourzac, 2017) is one
example of a hybrid memory, with DRAM and NVM connected to separate channels,
interfacing with a multi-core CPU chip using the JEDEC’s new NVDIMM
specification (jed, 2019). IBM POWER9 architecture (Sadasivam et al., 2017) is
another example, which uses embedded DRAM (eDRAM) as a write cache to NVM-
based main memory. Figure 1 illustrates both these hybrid architectures and we
evaluate them in Section 6.

Figure 1. DRAM-NVM hybrid memory architecture of (a) 3D XPoint Memory
(Bourzac, 2017) and (b) IBM POWER9 (Sadasivam et al., 2017).

Modern operating systems (OSes) such as Nimble (Yan et al., 2019) are already
aware of the performance and reliability asymmetry in hybrid memory. They
migrate write-intensive pages to DRAM (which has practically infinite
endurance) and read-intensive pages to NVM (which has a read latency
comparable to DRAM), starting from an initial random page placement
(Blagodurov et al., 2019). There are two key limitations in these OSes. First,
if pages are not placed in their matching memory (i.e., NVM or DRAM) at their
initial allocation, they can incur significant performance and energy overhead
during program execution due to the high bank and channel occupancy in moving
page data between the two memory. Second, limited write endurance is not the
only reliability issue in NVM. In fact, a recent study has shown that even
read accesses can lead to high-voltage related aging (another key reliability
issue) in a NVM’s peripheral circuit (Balaji et al., 2019).

Our objective is to improve performance and reliability (both endurance and
aging) of DRAM-NVM hybrid memory. We achieve this goal by exploiting the
following three major observations in this paper.

Observation 1: A significant number of pages are migrated more than once
during a program execution.

Figure 2 plots the fraction of memory pages with no migration, exactly one
migration, and more than one migration using Nimble’s dynamic page migration
policy for the evaluated workloads, which are detailed in Section 5.

Figure 2. Fraction of memory pages that suffer no migration, exactly one
migration, and more than one migration using Nimble for our evaluated
workloads.

We observe a wide variation in behavior across these programs. For instance,
over 93% of memory pages in perlbench are migrated at most once, whereas 95%
of all pages in roms are migrated more than once. On average, 67% of memory
pages in these programs suffer more than one migration during their execution.
These migrations lead to high energy and performance overhead.

Observation 1 leads to our first key idea that if a memory page is placed in a
matching memory during its initial allocation, many of these migrations can be
eliminated, leading to performance and energy improvements (Section 6). This
idea also leads to our next observation.

Observation 2: There are typically only a few first-touch instructions (FTIs)
in a program and only a small percentage of these instructions induce the most
memory accesses.

Modern OSes implement first-touch page allocation policy, where a virtual-to-
physical address translator allocates a random physical memory page from the
free pool to a virtual page address, when the virtual page is first touched by
a memory instruction in the program. We call this memory instruction first-
touch instruction (FTI).

Figure 3 plots the number of FTIs and referenced pages per billion
instructions of the evaluated workloads. We report total FTIs (outer first bar
in each set) and the number of FTIs that touch pages which serve over 90% of
memory accesses (inner first bar). We also report the pages referenced per
billion instructions (second bar). We observe that 1) there are very few FTIs
per billion instructions of each workload, and 2) on average, only 17% of FTIs
in a program touch pages which serve over 90% of memory accesses.

Figure 3. First-touch instructions (FTIs) per billion instructions of the
evaluated workloads.

Observation 2 leads to our second key idea of profiling FTIs based on the
number of accesses to memory pages they touch and using it to predict the
access intensity of a newly-referenced memory page, thereby placing it in a
matching memory during its initial allocation.

Observations 1 and 2 are related to OS-based page management in DRAM-NVM
hybrid memory. Our final observation is related to the internal architecture
of memory.

Observation 3: Parasitic components on a long bitline are a major source of
high latency in both DRAM and NVM, and a significant factor contributing to
high-voltage operations in NVM, which impact reliability.

This is observed and reported for DRAM (Lee et al., 2013). We expand this
observation for NVM, where each bit is represented by the resistance of a cell
(low resistance represents logic ‘1’ and high resistance logic ‘0’). An NVM
cell’s resistance is read or programmed by driving current through the cell
using a peripheral circuit, which consists of sense amplifiers (to read) and
write drivers (to write). We analyze the internal architecture of an NVM bank
and find that its peripheral circuit is several orders of magnitude larger
than the size of an NVM cell (Song et al., 2019; Cho et al., 2005; Dray and
Wei, 2018; Sandhu et al., 2020).222NVM, like DRAM, is organized
hierarchically. An example NVM of 128GB capacity can have 4 channels, with 4
ranks per channel, and 8 banks per rank. A bank can have 8 partitions, which
are similar to subarrays in DRAM (Kim et al., 2012). To amortize this large
size, memory designers connect a peripheral circuit to many (typically 4096)
NVM cells through a wire called a bitline. Numerous bitlines are laid in
parallel to form an NVM array (called a _tile_). A row of NVM cells is called
a _wordline_.

Figure 4 (a) illustrates an NVM tile with bitlines and wordlines. Many such
tiles make a partition (see our simulation parameters in Table 4). Figure 4(b)
illustrates the lumped RC circuit of a bitline to model its parasitic
components. The voltage drop (called the IR drop) on the bitline parasitic
needs to be compensated by a peripheral circuit to access the NVM cells on its
bitline. As we can see from this figure, farther a cell from the peripheral
circuit, higher is the IR drop.

Figure 4. (a) An NVM tile with bitlines and wordlines and (b) Lumped RC model
of a bitline.

Figure 5 plots the design analysis performed while architecting main memory.
We show such analysis for PCM, an emerging NVM, based on Micron’s 45nm design
(Bhattacharyya, 2019).333We expect the values to be of similar orders of
magnitude for other designs. The left y-axis plots the IR drop on a bitline
for SET, RESET, and READ operations as a function of the number of bitline
cells. The right y-axis plots the normalized cost per bit. We observe that the
normalized cost decreases as the number of bitline cells increases. However,
higher the number of bitline cells, higher is the IR drop. The trade-off point
is typically set to 4096 cells in most PCM designs (Bhattacharyya, 2019; Lung
et al., 2016; Redaelli and Perrone, 2017; Villa, 2018). Similar analysis
conducted on Micron’s 45nm DRAM design suggests that 512 cells per bitline in
a DRAM subarray gives the best latency and cost trade-offs (Lee et al., 2013;
Das et al., 2018). In this paper, we assume, without loss of generality, each
bitline in NVM and DRAM contains 4096 and 512 cells, respectively.

Figure 5. Architecting the number of PCM cells per bitline.

Table 1 reports the biasing voltage needed to access the nearest cell
(1${}^{\text{st}}$ cell), the farthest cell (4096${}^{\text{th}}$ cell) and an
intermediate cell (512${}^{\text{th}}$) on a bitline in PCM. Higher voltages
are needed to access cells that are farther from their peripheral circuit.
This has two implications. First, cells that are nearer to their peripheral
circuit can be accessed faster. This is because the on-chip voltage regulator,
which supplies the biasing voltage for a peripheral circuit, has faster
response time and higher energy efficiency to generate lower voltages. Second,
operating a peripheral circuit at a lower voltage incurs lower circuit aging,
which improves reliability (see our reliability formulation in Section 4.1.2).
We conclude that PCM (and in general, NVM) has asymmetric latency and
reliability in accessing its content.

Table 1. PCM’s biasing voltages. Cell Op. | Bias Voltage
---|---
Nearest cell | Farthest cell | Intermediate cell
(1${}^{\text{st}}$ cell) | (4096${}^{\text{th}}$ cell) | (512${}^{\text{th}}$ cell)
SET | 2.1 | 3.7V | 2.3V
RESET | 6.8 | 7.1V | 6.9V
READ | 0.96 | 2.85V | 1.2V

Observation 3 leads to our third key idea of introducing an isolation
transistor on each bitline in DRAM and NVM, to allow its length to appear
shorter when accessing cells nearer to its peripheral circuit, thereby
achieving low latency in DRAM and NVM and additionally, high reliability in
NVM. Segmented bitlines create latency and reliability asymmetry, i.e., tiers
within both DRAM and NVM.

We introduce MNEME444In Greek mythology, MNEME is the muse of memory. MNEME
means persistent effect of memory of past events, which are the first-touch
instructions in the context of this paper., a mechanism that builds on the
three ideas above to enable additional tiers in hybrid memory and exploit
these tiers through an efficient OS-level page allocation policy. MNEME places
a newly-referenced memory page to the best tier during its initial allocation,
minimizing channel and bank occupancy associated with migration of page data
during execution. Through MNEME, we make the following key contributions.

  * •

We introduce a new memory architecture with segmented bitlines within DRAM and
NVM to improve performance and reliability during data accesses.

  * •

We propose an approach to predict access intensity of a newly-referenced
memory page using the page’s first-touch instruction (FTI) and place it in a
matching memory tier during its initial allocation, reducing page migrations
during program execution.

  * •

We develop our FTI-based page allocation for the entire program duration to
adapt to and make correct allocation decisions for different phases of
execution in a program with potentially distinct working sets.

  * •

We show how to reduce channel occupancy during page migration between tiers of
the same memory, thereby improving performance.

  * •

We introduce an efficient hardware implementation of MNEME using Bloom
filters.

  * •

We implement MNEME for two hybrid memory architectures and also for commodity
DRAM-based mainstream architecture, and show significant performance and
reliability improvements for both single-program and multi-programmed
workloads.

##  2\. New Segmented Bitline Architecture of MNEME

Figure 6(a) shows the proposed segmented bitline architecture, where each long
bitline in DRAM and NVM is split into two segments using an isolation
transistor: the segment connected directly to the peripheral circuit is called
the near segment, whereas the other is called the far segment. Cells in the
near segment can be accessed faster using lower bias voltages due to the
reduced parasitic on the current path (see Figure 6(b)). This improves
performance. Additionally, by using lower voltages, circuit aging when
accessing the near segment is minimized, which improves reliability. Aging-
related reliability is particularly critical for NVM, which requires higher
operating voltages than DRAM (Song et al., 2020).

Figure 6. (a) Bitlines partitioned into segments. (b) Accessing a near segment
cell. (c) Accessing a far segment cell.

However, the isolation transistor increases access latency of the far segment
and introduces its ON resistance in the current path, which imposes additional
bias requirement for the peripheral circuit (see Figure 6(c)). This lowers
reliability of the peripheral circuit in accessing the far segment.

Segmented bitlines is previously proposed for DRAM subarrays (Lee et al.,
2013) and they lead to performance trade-offs in accessing near versus far
segments. We propose segmented bitlines for DRAM-NVM hybrid memory, which
introduces reliability trade-off, in addition to the performance ones.

To evaluate the performance improvement using this new memory architecture,
Figure 7 plots the execution time of 15 workloads (see Section 5) on a hybrid
memory with segmented bitlines. Results are normalized to the execution time
of a Baseline design, where bitlines are not segmented. We observe that simply
introducing memory tiers by creating segments in each bitline is just not
enough to guarantee performance improvement; in fact, performance improves by
only 2% on average for these workloads. We believe that our hybrid tiered-
memory architecture can only deliver on its promises if the inter- and intra-
memory asymmetries are exploited efficiently by an operating system (OS)-level
page management policy, which we introduce next.

Figure 7. Execution time of our evaluated workloads normalized to Baseline
where bitlines are not segmented.

Our hybrid tiered-memory architecture is shown in Figure 8, where the memory
tiers are arranged with increasing access latency from the CPU. The figure
also shows how our architecture differs from the two state-of-the-art
approaches: TL-DRAM (Lee et al., 2013), which only uses memory tiers within
DRAM-based main memory and Nimble (Yan et al., 2019), which uses DRAM-NVM
hybrid main memory like ours but the bitlines are not segmented. We evaluate
both these state-of-the-art approaches in Section 6.

Figure 8. Proposed hybrid tiered-memory architecture.

##  3\. New Page Management Policy of MNEME

Figure 9 shows a high-level overview of our page allocation policy to exploit
our tiered architecture. A program is broken down into fixed intervals (called
_phases_). At each phase, we profile FTIs based on accesses to pages they
touch. This information is then used to decide the initial placement of all
newly-referenced memory pages of the subsequent phases.

Figure 9. Proposed program execution.

Figure 10 shows an example of using FTIs to predict the access intensity of
newly-referenced memory pages in MNEME.

Figure 10. Examples of subsequent memory accesses created by first-touch load
and store instructions.

Assume that no profile information is available in the beginning. The program
counter a1 causes the initial page fault, loading Page $A$ into the far memory
segment. During the course of program execution, the load instruction at a2
causes a page fault, loading page $E$ into the far memory segment. Similarly,
the store instruction at a3 also loads page $Q$ into the far segment. The
instructions at addresses, a1, a2, and a3 are the FTIs. Now assume that as we
iterate through the for loop, the load and store instructions, located at
addresses a2 and a3, respectively, generate numerous accesses to $E$ and $Q$,
respectively. MNEME records them as FTIs that induce large numbers of accesses
to any page that these instructions might load in the future. Therefore, later
on when the load instruction references an address that requires page $F$ to
be loaded, MNEME will load this page into the near memory segment. Similarly,
$R$ will also be loaded in the near segment when accessed by the store
instruction.

In addition to load and store statements, implementations of branch or jump
tables will also contain FTIs that can be predicted to load frequently-
accessed pages; for example, when a jump instruction is the FTI that causes
the page containing the corresponding function to be loaded. Fig. 3 shows that
such FTIs are a major source of memory references.

A conceptual overview of MNEME is shown in Figure 11. At a high-level, the
memory controller maintains a table containing the memory addresses of access-
intensive first-touch instructions (i.e., their program counter value). We
call this FTI table. We split the execution of an application into phases,
with each phase comprising of 100 million instructions (See Section 6.6 for
evaluation on the size of an execution phase).

Figure 11. A conceptual overview of MNEME.

To allocate a newly referenced memory page in an execution phase, the OS page-
fault handler runs a custom instruction to check if the virtual address
corresponding to the FTI of the page hits in the FTI table. If a match is
found, the memory page is predicted to be access-intensive. The page-fault
handler allocates this new memory page to the near memory segment. We explain
later how to choose between DRAM and NVM. The memory controller then uses
reduced timing and voltage parameters to access this page, improving
performance and reliability. Otherwise, the FTI is considered to be unknown
and possibly referencing a non-access intensive memory page. The OS page
handler allocates the memory page to the far segment, while tracking the
number of accesses this page generates within the phase, leveraging OS page
tracking structures, and recording it inside a table. We call this Access
Intensity Record (AIR). At the end of each phase, the top access-intensive
unknown FTIs of AIR (with number of accesses higher than a threshold) are
inserted into the FTI table to predict and place all new memory pages to near
or far segments. In this way, the FTI table is constantly updated with new
access-inducing FTIs that are uncovered during program execution.

Using the FTI table and AIR, MNEME can place a new memory page to a specific
segment in DRAM or NVM. To select the specific memory type (i.e., DRAM vs.
NVM), we introduce the following changes: 1) we maintain two FTI tables: one
holding those FTIs that touch more write-intensive pages (we call this
_FTI_W_), and another holding those FTIs that touch more read-intensive pages
(we call this _FTI_R_), and 2) extend the AIR to record the number of read-
inducing and write-inducing pages that each FTI touches.

For a new memory page in a program phase, there can be four possibilities with
the corresponding FTI.

  * •

_Hit in FTI_W and miss in FTI_R:_ the FTI is predicted as write-access
inducing. So, allocate the memory page to a near segment in DRAM.

  * •

_Miss in FTI_W and hit in FTI_R:_ the FTI is predicted as read-access
inducing. So, allocate the memory page to a near segment in NVM.

  * •

_Hit in FTI_W and hit in FTI_R:_ the FTI is predicted as both read and write
inducing. So, allocate the memory page to a near segment in DRAM
(conservative).

  * •

_Miss in FTI_W and miss in FTI_R:_ the FTI is predicted as non-access
inducing. So, allocate the memory page this FTI touches to a far segment in
NVM if space is available there, otherwise allocate it to a far segment in
DRAM. Additionally, make an entry for the FTI in AIR and start recording
accesses to the page.

If MNEME predicts the access intensity of a new memory page correctly (which
we evaluate in Section 6), the page will be placed in the correct memory tier
during its initial allocation, reducing run-time page migration overhead.
Otherwise, the page will be placed in an incorrect tier and will be migrated
by tracking its accesses during program execution.

Page migration in MNEME: Figure 12 shows the journey of hot and cold pages
through tiers of the proposed hybrid tiered-memory architecture. MNEME
supports two types of migrations: 1) page migrations between tiers of the same
memory unit, and 2) page migrations across tiers of different memory units.
For within memory migrations, we propose an approach where a page can be
migrated from one tier to another in the same memory bank without utilizing
the memory channels. This can be achieved for DRAM using two back-to-back
activates (see Sec. 4.1.1) utilizing row buffers, which are shared between
read and write operations (see RowClone (Seshadri et al., 2013) for instance).

Figure 12. Data migration in MNEME.

However, for PCM, and NVM in general, this is not straight forward because the
peripheral circuit consists of separate hardware to read and write. Figure 13
shows the architecture of a peripheral circuit in an NVM (e.g., PCM) bank
(Song et al., 2019). The peripheral circuit consists of the sense amplifier
(to read) and the write driver (to write), which are connected to a bitline
using transistors M1 and M2. From the write driver’s internal circuit diagram
shown in Figure 13, we observe that the write driver can be viewed as a
collection of two components – the write pulse shaper logic, which generates
the current pulses necessary for the cell’s SET and RESET operations, and the
verify logic, which verifies the correctness of these operations. These two
circuit components together serve write requests from the bank using a write
scheme known as _program-and-verify_ (P&V) (Nirschl et al., 2007; Goda et al.,
2019).

Figure 13. Internal circuit of a bank’s peripheral structure.

Based on this observation, we propose _simple_ circuit modifications to
introduce the decoupling transistor M (see Fig. 13), which can be configured
when needed, to transfer data from the sense amplifier to the verify logic. As
a result of this modification, we can program the data read in the sense
amplifier to a different row in the bank using the write driver. This
facilitates data migration between tiers of the same memory bank without
utilizing external memory channels.

To migrate pages across tiers of different memory units, we still use the
memory channel, which leads to performance overhead. However, our OS-level
page allocation policy minimizes these migrations significantly (see Section
6.5).

##  4\. Implementation of MNEME

MNEME consists of two key components: 1) interface to support efficient data
tiering within and across memory units in hybrid memory, and 2) intensity
prediction via FTI table and AIR. We discuss how to implement each of these
components in order to design an efficient implementation of MNEME.

###  4.1. Interface to Support Tiered Hybrid Memory

Figure 14 shows the handshaking between OS, CPU and memory, via the memory
controller. Inside the memory controller, there is a separate read and write
queue to buffer requests to the memory. The scheduler schedules requests from
these queues using its access scheduling policy. We use the FR-FCFS policy
(Rixner et al., 2000), where the scheduler prioritizes requests that hit in
the row buffer in a memory bank.

Figure 14. A full-system overview.

The address mapping block is responsible for selecting the desired timing and
voltage parameters based on the memory segment (near or far) that a request
hits. To explain this, we use an example of 128GB NVM with 4 channels, 4 ranks
per channel, 8 banks per rank, 8 partitions per bank, 128 tiles per partition,
4096 wordlines and 2048 bitlines per tile. Considering bank interleaving, the
address mapping scheme is as follows: [36:35] = rank address, [34:32] =
partition address, [31:25] = tile address, [24:13] = row address, [12:11] =
column address, [10:8] = bank address, [7:6] = channel address, and [5:0] =
byte address. We assume, without loss of generality, that an isolation
transistor divides each bitline into near segment with 512 cells and far
segment with 3584 cells (= 4096 - 512).

To decode the segment that a request hits, we use bit slicing on the wordline
address bits [24:13]. Therefore, address bit 22 is used as the segment select
bit (‘0’$\implies$ near segment and ‘1’$\implies$ far segment). The segment
select bit is used to select segment-specific timing parameters stored in a
lookup table (LUT) inside the memory controller. The memory request and the
selected timing parameters are forwarded to a command generator, implemented
as a state machine, which issues memory-specific commands at appropriate
intervals.

The description above applies to DRAM as well, with the exception of the on-
chip voltage regulator, which is needed only for NVM to drive current through
its cells. Furthermore, we use the DRAM configuration of Lee et al. (Lee et
al., 2013), where a bitline contains 512 cells and is divided using an
isolation transistor into near segment with 128 cells and far segment with 384
cells. Bit slicing is performed accordingly.

We now provide latency and reliability analysis of near and far segments in
DRAM and NVM. We consider the DRAM architecture of Lee et al. (Lee et al.,
2013) and the PCM architecture of Redaelli (Redaelli, 2018), both from Micron.

####  4.1.1. Latency Analysis

To understand the latency impact due to bitline segmentation, we briefly
review memory-timing parameters. The following discussion applies for both
DRAM and PCM. To serve a memory request that accesses data at a particular row
and column address within a bank, a memory controller issues _three_ commands
to the bank.

  * •

ACTIVATE: activate the wordline and enable the peripheral circuit for the
memory cells to be accessed.

  * •

READ/WRITE: drive read or write current through the cell (PCM) or share charge
from the cell (DRAM). After this command executes, the data stored in the cell
is available at the output terminal of peripheral circuit, or the write data
is programmed to the cell.

  * •

PRECHARGE: deactivate the wordline and bitline, and prepare the bank for the
next access.

Figure 15. Memory timings for read requests.

Figure 15 shows different memory timing parameters when serving two read
requests. Table 2 reports these timing parameters for the near and far segment
of DRAM and PCM. The parameters for DRAM are obtained from Lee et al. (Lee et
al., 2013), scaled to 45nm technology nodes using predictive technology
scaling (Cao and McAndrew, 2007). The timing parameters for PCM are obtained
via SPICE simulations (Antognetti and Massobrio, 1990) with 45nm PDK (Stine et
al., 2007). Table 2 is stored in a LUT in our memory controller.

Table 2. Latency incurred by read and write requests, respectively, to near and far segments of DRAM and PCM. | tRCD | tCL | tBL | tRP | tRC
---|---|---|---|---|---
DRAM | near | Read | 9.3ns | 5.5ns | 7.5ns | 5.5ns | 27.8ns
Write | 9.3ns | 5.5ns | 7.5ns | 5.5ns | 27.8ns
far | Read | 15ns | 15ns | 7.5ns | 15ns | 52.5ns
Write | 15ns | 15ns | 7.5ns | 15ns | 52.5ns
PCM | near | Read | 3.75ns | 22.5ns | 15ns | 0ns | 41.25ns
Write | 3.75ns | 101ns | 15ns | 0ns | 119.75ns
far | Read | 3.75ns | 37.5ns | 15ns | 0ns | 56.25ns
Write | 3.75ns | 142.8ns | 15ns | 0ns | 161.55ns

####  4.1.2. Reliability Analysis

Table 3 summarizes the sources of reliability concerns in NVM. In this work,
we consider two dominant reliability issues in PCM: 1) finite endurance of PCM
cells and 2) high voltage-related aging of CMOS devices in a peripheral
circuit. We formulate these next.

Table 3. Reliability issues in NVM. Reliability Issues | NVM
---|---
High-voltage related circuit aging | PCM, Flash
High-current related circuit aging | OxRAM, STT-MRAM
Read disturbance | All
Limited endurance | All

_Endurance-related lifetime :_ Endurance-related lifetime depends on: 1) how
many times a PCM cell can be programmed ($N_{e}$) and 2) how frequently the
cells are programmed ($N_{f}$) (Qureshi et al., 2012). If $N_{WL}$ is the
total number of wordlines in a PCM bank, the endurance-related lifetime can be
estimated as

(1) $\footnotesize{L_{e}}=N_{WL}*N_{e}/N_{f}.$

_Aging-related lifetime :_ High-voltage operations lead to reliability issues
such as negative-bias temperature instability (NBTI), hot carrier injection
(HCI), and time-dependent dielectric breakdown (TDDB) (Das et al., 2015;
Balaji et al., 2019; Das et al., 2013a, b, 2014a, 2014b; Das and Kumar, 2012;
Srinivasan et al., 2004; Das et al., 2014c). We illustrate NBTI, which is a
dominant reliability issue in scaled technology nodes. NBTI-induced aging of a
CMOS device in a peripheral circuit at temperature $T$ is calculated as

(2) $\footnotesize\mathcal{A}(T)=\sum_{i=0}^{N_{a}-1}g_{0}(T)\cdot
V_{\text{bias}}^{a}\cdot tRC^{b},$

where $N_{a}$ is the number of PCM accesses, $g_{0}(T)$, $a$, and $b$ are
material-dependent constants (Balaji et al., 2019). The bias voltage for a
specific PCM operation, $V_{\text{bias}}$, is obtained from Table 1, and the
PCM timing parameter, $tRC$, from Table 2.

Using Equation 2, the reliability ($R(T)$) and lifetime ($L_{a}$) can be
computed as

(3) $\footnotesize R(T)=e^{-\mathcal{A}(T)^{\beta}}\text{ and }L_{a}=\int
R(T).$

From Equations 2 and 3, we can conclude that aging can be used as a measure of
lifetime. In Section 6, we show improvements of MNEME in terms of $L_{e}$ and
$\mathcal{A}$.

###  4.2. Efficient Implementation of FTI Table and AIR

We now discuss an efficient implementation of MNEME.

####  4.2.1. Implementing FTI Table

MNEME stores most access-inducing FTIs (i.e., their program counters) from an
execution phase in the FTI Table. A naive way to implement the FTI table is to
use a row for each FTI. However, the exact number of rows within this table
will vary depending on the number of FTIs, which is program-specific. If a
table’s capacity is inadequate to store all high-access inducing FTIs, MNEME
will start predicting every new FTI as non access-intensive, once the table is
full. The OS page fault handler will then allocate all new pages to the far
memory segment, providing no significant performance improvement. Therefore,
the FTI table must be sized conservatively (i.e. assuming the maximum number
of FTIs), leading to a large hardware cost for table storage and lookup. We
propose to use Bloom filer to implement the FTI table.

The _Bloom filter_ is a memory-efficient data structure to represent set
membership (Bloom, 1970). The price paid for this efficiency is that this
filter is a probabilistic data structure: it tells if an element is either
definitely not in the set (zero false positive) or may be in the set (non-zero
false negative).

Figure 16. Operation of a Bloom filter.

The filter is implemented as a bit array of length $m$ with $k$ distinct hash
functions. Figure 16 shows a filter in which $m=16$ and $k=3$. To insert an
element $x$ in the filter, it is hashed with all three functions, and all of
the bits in the corresponding positions are set to 1. Conversely, to test if
an element $y$ is in the filter, it is again hashed using all three functions;
and if all of the bits at the corresponding bit positions are 1, the element
is declared to be present in the Bloom filter. In Fig.16, $y$ is declared to
be not present.

Because bits are never reset: 1) an element once inserted cannot be deleted
from the filter555Certain modifications to the Bloom filter allow for deletion
of elements. One example is the Cuckoo filter (Fan et al., 2014)., and 2) a
false negative can never occur. The rate of false positive is approximately

(4) $\footnotesize\left(1-e^{-kn/m}\right)^{k},$

where $n$ is the number of elements expected to be inserted in the filter.
MNEME uses two Bloom filters for the FTI_R and FTI_W tables, each implemented
with $m=128$, $k=3$.

####  4.2.2. AIR Implementation

The AIR is implemented as a table with $D$ rows, each having five fields: a
valid field, the program counter value of the FTI, the number of read and
write inducing pages touched by this FTI, and the number of accesses that go
to these pages. Within a program phase, the least-frequently used entry in the
AIR is overwritten with a new FTI; the field recording the total number of
accesses is used as the frequency estimate. Upon completion of a phase, any
entry having frequency higher than a threshold is inserted into the Bloom
filter. Subsequently, the AIR is reset by setting the valid field of all its
entries to 0.

##  5\. Evaluation Methodology

To evaluate MNEME, we develop a cycle-accurate DRAM-PCM hybrid memory
simulator with the following:

  * •

A Cycle-level x86 multi-core simulator, whose front-end is based on Pin (Luk
et al., 2005). We configure this to simulate 8 out-of-order cores.

  * •

A main memory simulator, closely matching the JEDEC Nonvolatile Dual In-line
Memory Module (NVDIMM)-N/F/P Specifications (jed, 2019). This simulator is
composed of Ramulator (Kim et al., 2016), to simulate DRAM, and a cycle-level
PCM simulator based on NVMain (Poremba et al., 2015).

  * •

Power and latency for DRAM and PCM are based on Intel/Micron’s 3D Xpoint
specification (Bourzac, 2017; Redaelli, 2018). Energy is modeled for DRAM
using DRAMPower (Chandrasekar et al., 2012) and for NVM using NVMain with
parameters from (Redaelli, 2018).

Table 4 summarizes the various simulation parameters.

Table 4. Major simulation parameters. Processor |  8 cores, 3 GHz, out-of-order
---|---
L1-I/D cache |  Private 64KB per core, 4-way
L2 cache |  shared, 4MB, 8-way
DRAM Main Memory |  64GB, Micron DDR3
|  2 channels, 4 ranks/channel, 8 banks/rank, 128 sub-arrays/bank, 512
rows/sub-array
|  Memory clock = 1066MHz
|  Near bitline segment = 128 cells
|  Far bitline segment = 384 cells (= 512 -128)
PCM Main Memory |  128GB, Micron DDR3 (Redaelli, 2018)
|  4 channels, 4 ranks/channel, 8 banks/rank, 8 partitions/bank, 128
tiles/partition, 4096 rows/tile
|  Memory clock = 1066MHz
|  Near bitline segment = 512 cells
|  Far bitline segment = 3584 cells(= 4096 -512)

We evaluate the following main memory architectures.

  * •

M1: DRAM-PCM hybrid memory with DRAM and PCM placed on separate DIMMs, sharing
a common main memory address space. This is the primary hybrid memory
architecture that we evaluate in this paper, and is similar to Intel Optane
(Bourzac, 2017) using PCM instead of SSD.

  * •

M2: PCM main memory with DRAM as write cache. This is similar to the
architecture of IBM Power9 (Sadasivam et al., 2017).

  * •

M3: Mainstream DRAM-based main memory architecture similar to Intel Skylake
(Doweck et al., 2017).

We evaluate architectures M2 and M3 to show that MNEME improves performance of
other memory architectures.

We evaluate the following techniques.

  * •

Baseline allocates a page randomly to a free physical address. Pages are not
migrated between DRAM and PCM during program execution. Bitlines are not
segmented.

    * –

for M1, the Baseline is Intel Optane (Bourzac, 2017).

    * –

for M2, the Baseline is IBM Power9 (Sadasivam et al., 2017).

    * –

for M3, the Baseline is Intel Skylake (Doweck et al., 2017).

  * •

Nimble (Yan et al., 2019) supports M1-type hybrid memory. It migrates pages
between DRAM and PCM during program execution, starting from a random physical
address allocation. Bitlines are not segmented.

  * •

TL-DRAM (Lee et al., 2013) supports DRAM (M3). It uses the page management
policy of Baseline. Each bitline in a DRAM bank is partitioned into near and
far segments.

  * •

MNEME supports both M1 and M2-type hybrid memory architectures. It 1) uses
segmented bitlines for DRAM and PCM, 2) controls initial page allocation to
correct memory tiers, and 3) minimizes channel occupancy during page
migrations between tiers of the same memory, to improve performance and
reliability.

We evaluate all single-core and multi-programmed workloads from the SPEC
CPU2017 suite (Bucek et al., 2018). Table 5 reports the workloads that we
present in Section 6. These workloads are chosen because they have at least 1
cache Miss Per Kilo Instructions (MPKI) (see Fig. 17). For other workloads
with low MPKI (those not presented in Sec. 6), MNEME neither significantly
improves nor hurts performance and reliability.

Table 5. Evaluated workloads. single-core |  8 copies each of blender, bwaves, cactuBSSN, cam4, gcc, imagick, nab, namd, omnetpp, perlbench, povray, roms, wrf, xalancbmk, xz
---|---
multi-programmed |  MP1 (2 copies each of blender, bwaves, cactuBSSN, and cam4) and MP2 (2 copies each of perlbench, wrf, xalancbmk, and xz)

Figure 17. MPKI for our evaluated workloads.

All workloads are executed for 10 billion instructions.

##  6\. Results and Discussions

###  6.1. Summary of Key Results

Table 6 summarizes MNEME’s improvements.

Table 6. Summary of key results. | System | Energy | Migration | Lifetime
---|---|---|---|---
| Perf. | Consump. | Overhead | (Sec. 6.9)
MNEME vs. | (Sec. 6.2) | (Sec. 6.8) | (Sec. 6.5) | Endurance | Aging
Intel Optane | $21\%\uparrow$ | $19\%\downarrow$ | – | – | –
Nimble (Yan et al., 2019) | $16\%\uparrow$ | $18\%\downarrow$ | $71.2\%\downarrow$ | $20\%\uparrow$ | $33\%\downarrow$
IBM Power9 | $15\%\uparrow$ |
Intel Skylake | $15\%\uparrow$ |
TL-DRAM (Lee et al., 2013) | $13\%\uparrow$ |

###  6.2. Overall System Performance

We report overall system performance for three configurations: 1) _M1:_ DRAM-
PCM hybrid memory with DRAM and PCM on separate DIMMs, 2) _M2:_ DRAM-PCM
hybrid memory with DRAM as write cache to PCM, and 3) _M3:_ DRAM-based system.

####  6.2.1. Hybrid Main Memory Architecture M1

Figure 18 reports the execution time of each workload for our evaluated
systems normalized to Baseline. The simulator is configured for our primary
DRAM-PCM hybrid main memory architecture with a common address space. We make
the following three main observations.

Figure 18. Execution time, normalized to Baseline for DRAM-PCM hybrid main
memory with shared address.

First, Nimble achieves better performance than Baseline by an average of 7%
due to Nimble’s policy to migrate hot pages from PCM to DRAM, which reduces
execution time (DRAM has lower access latency than PCM). Second, for workloads
such as blender and bwaves, performance of Nimble is, in fact, worse than
Baseline because of the high overhead of page migrations in Nimble. Third,
MNEME’s performance is the best among all three systems. On average, the
execution time of MNEME is 21% lower than Baseline and 16% lower than Nimble.
This improvement is due to 1) MNEME’s segmented bitline architecture and 2)
MNEME’s intelligent initial page allocation policy to exploit performance
asymmetries in memory tiers.

####  6.2.2. Hybrid Main Memory Architecture M2

Figure 19 reports the execution time of each workload for our evaluated
systems normalized to Baseline with the simulator configured for DRAM-PCM
hybrid main memory with DRAM configured as write cache to PCM. We make the
following two main observations.

Figure 19. Execution time, normalized to Baseline for DRAM-PCM hybrid memory
with DRAM as write cache.

First, using the proposed segmented bitline architecture, performance of
Baseline improves only marginally, by an average of 2% (first bar in each
set). See also Observation 1. Second, MNEME’s performance is the highest among
all three systems. On average, MNEME’s execution time is 15% lower than
Baseline and 14% lower than segmented bitlines.

####  6.2.3. DRAM-based Main Memory Architecture M3

Figure 20 reports the execution time of each workload for our evaluated
systems normalized to Baseline with the simulator configured for DRAM-based
main memory. We observe that performance of TL-DRAM is marginally better than
Baseline. MNEME has the highest performance among all three systems. On
average, MNEME’s execution time is 15% lower than Baseline and 13% lower than
TL-DRAM.

Figure 20. Execution time, normalized to Baseline for DRAM-based main memory.

Unless otherwise stated, following results are for our primary memory
architecture, i.e., DRAM-PCM hybrid memory with a common memory address space
(M1).

###  6.3. Multi-Programmed Workloads

Figure 21 plots the execution time of MNEME normalized to Nimble for 2 multi-
programmed workloads on 2-core (2-channel), 4-core (4-channel), and 8-core
(8-channel) systems.

Figure 21. Execution time normalized to Nimble for multi-programmed workloads
on 2-core, 4-core, and 8-core.

We observe that for 2, 4, and 8 cores in the system, MNEME provides 18%, 24%
and 31% performance improvement for MP1, and 26%, 36% and 38% performance
improvement for MP2, compared to Nimble. The performance improvement of MNEME
increases with increasing number of channels. This is because, with more
channels, bank access latency becomes the primary performance bottleneck.
Therefore, MNEME, which reduces the average bank access latency, provides
better performance with more channels.

###  6.4. Memory Access Distribution

Figure 22 plots the memory accesses to near and far memory segments of each
workload for Nimble and MNEME. We make the following two main observations.

First, on average, only 13% of accesses go to near segments in memory banks
using the initial page allocation and hot page migration policy of Nimble.
Second, MNEME directs an average of 64% of accesses to near memory segments
using its intelligent initial page allocation policy. This leads to
significant performance improvement (see Section 6.2).

Figure 22. Memory accesses to near and far segments.

###  6.5. Migration Overhead

Figure 23 plots the page migration-related accesses in MNEME normalized to
Nimble for each workload. We observe that page migration-related accesses in
MNEME are lower than Nimble by an average of 71.2%. This reduction is because
1) MNEME places new pages in correct memory tiers during their initial
allocation using access profiles of observed first-touch instructions in the
program, which reduces the average number of inter-memory page migrations, and
2) MNEME uses its new peripheral circuit design to eliminate channel usage for
page migrations within each memory bank.

Figure 23. Migration-related accesses normalized to Nimble.

###  6.6. Length of Execution Phases

Figure 24 reports the execution time of MNEME normalized to Nimble for each
workload. The first bar in each set is for the default phase length of 100
million instructions. The second and third bars are for phase length of 250
million and 500 million instructions, respectively. We observe the execution
time of MNEME to increase with the length of the phase interval. This is
because, lower phase intervals allow finer control of page allocation,
resulting in higher performance (i.e., lower execution time) than Nimble.
However, lower phase intervals also result in higher overhead due to 1)
frequent updates to FTI Table and AIR and 2) frequent page migrations and
updates to page table, impacting performance. Phase interval of 100 million
instructions gives the best performance and overhead trade-off.

Figure 24. Execution time, normalized to Nimble for program phase intervals of
100M, 250M, and 500M instructions.

###  6.7. FTI-based Access Intensity Prediction

Figure 25 illustrates how MNEME improves its page allocation decisions over
time, improving overall performance. The bottom subfigure shows the increase
in stored FTIs during the execution of cam4. The top subfigure shows the
fraction of memory pages (i.e., their program counters) that hit in the FTI
Table. We observe that cam4 undergoes a change in behavior after executing
$\approx 5.5$ billion instructions and then again after $\approx 8.7$ billion
instructions, due to potentially distinct work sets. We see a dip in the
number of pages that hit in the FTI Table (top subfigure). Therefore, the
number of stored FTIs increases sharply around these time (bottom subfigure)
because MNEME starts inserting the newly observed FTIs into the FTI table to
improve its allocation decisions. This results in an increase in the number of
page hits (top subfigure) during subsequent execution of cam4.

Figure 25. Illustration of the FTI-based access intensity prediction for cam4.

###  6.8. Energy Consumption

Figure 26 reports the total energy consumption (demand accesses and page
migrations) of each workload for our evaluated systems normalized to Baseline.
We observe that Nimble has lower energy consumption than Baseline by an
average of only 2%. Although energy consumption of demand accesses is lower in
Nimble, the potential energy savings are overshadowed by the page migrations.
MNEME has the lowest energy consumption (on average, 19% lower than Baseline
and 18% lower than Nimble). These savings are achieved in MNEME because it
reduces page migrations on the memory channel significantly by 1) initially
allocating a page to a correct tier, and 2) facilitating inter-segment data
transfers, which reduce channel occupancy.

Figure 26. Energy consumption, normalized to Baseline.

###  6.9. Reliability

We evaluate two reliability issues: endurance and NBTI.

####  6.9.1. Endurance-related Lifetime

Figure 27 reports the endurance-related lifetime (computed using Equation 1)
of each workload for our evaluated systems normalized to Nimble. We make the
following observation.

Figure 27. Endurance lifetime, normalized to Nimble.

MNEME improves endurance-related lifetime by an average of 20% compared to
Nimble. This improvement is because of the extra tiers that MNEME creates
inside each memory using isolation transistors. Pages that are not frequently
referenced can now stay in DRAM far segments for sometime, before they are
migrated to PCM. This reduces PCM writes, which improves endurance.

####  6.9.2. NBTI-related Aging

Figure 28 reports the NBTI-related aging (computed using Equation 2) of each
workload for our evaluated systems normalized to Nimble. We observe that MNEME
has 33% lower NBTI-related aging than Nimble. This is because MNEME uses lower
bias voltages to access near PCM segments (Table 1) due to its segmented
bitline architecture. Also, the CMOS devices in PCM’s peripheral circuit are
stressed for reduced time duration than Nimble due to lower timing
requirements of the near segments (Table 2). Both these factors contribute to
lower NBTI aging (Balaji et al., 2019).

Figure 28. NBTI-related aging, normalized to Nimble.

###  6.10. Design Area Analysis

MNEME introduces two changes: increase in memory die size due to isolation
transistors and overhead in the memory controller to maintain FTI Table and
AIR.

####  6.10.1. Area Overhead on Memory-side

Adding an isolation transistor to each bitline increases the area of each
bank. We estimate this for DRAM and PCM as follows. In DRAM, the sense
amplifier and the isolation transistor are respectively 115.2x and 11.5x
taller than an individual DRAM cell. For a subarray of 512 DRAM cells per
bitline, the area overhead is $\frac{11.5}{115.2+512}=1.83\%$ (Lee et al.,
2013).

In PCM, the peripheral circuit and the isolation transistor are respectively
384x and 9.6x taller than an individual PCM cell. For a PCM partition of 4096
PCM cells per bitline, the area overhead is $\frac{9.6+9.6}{384+4096}=0.43\%$
(including the change to support in-memory migrations).

####  6.10.2. Area Overhead on CPU-side

FTI Tables are implemented as two 128-bit Bloom filters with a total size of
256 bits. AIR is implemented as an 8-entry table with 1-bit valid field, a
32-bit field for the program counter, a 32-bit field for counting accesses,
and two 16-bit fields for counting pages. The total area overhead of AIR is
97B. The LUT stores 4 extra rows in Table 2 (2 for PCM read and write latency
of near segment and 2 for DRAM read and write latency of near segment). The
extra area overhead is 160 bits (= 4 * 5 entries per row * 8 bits per entry).
So, MNEME introduces a total of 149 bytes in storage in the memory controller,
corresponding to an area overhead of $6\times 10^{-4}\text{mm}^{2}$ at 45nm.
Given the cost sensitivity of memory designs, designers can still benefit from
MNEME’s standalone page allocation policy, without segmented bitlines.

Figure 29 plots the execution time of each workload for Nimble and MNEME,
normalized to Baseline. The simulator is configured for DRAM-PCM hybrid memory
architecture with shared address space. Bitlines are not segmented either in
DRAM or in PCM. We observe that MNEME’s performance is still better. On
average, MNEME’s execution time is 15% lower than Baseline and 8% lower than
Nimble.

Figure 29. Execution time, normalized to Baseline for mainstream memory
without segmented bitlines.

##  7\. Related Works

To our knowledge, this is the first work that 1) enables segmented bitline
architecture for non-volatile memory and analyzes its performance and
reliability impacts, 2) develop a strategy to intelligently place pages in
near and far segments of different memory units during their initial
allocation, reducing run-time migration overhead, and 3) introduces page
migrations within tiers of the same memory, without moving data on the memory
channels.

###  7.1. Performance, Energy, and Endurance Optimizations

Many prior works optimize performance, energy, and endurance of PCM (Lee et
al., 2010b; Yoon et al., 2015; Lee et al., 2010c; Qureshi et al., 2009, 2012;
Arjomand et al., 2016; Cho and Lee, 2009; Seong et al., 2010; Akram et al.,
2018; Song et al., 2020, 2019). Song et al. propose exploiting partition-level
parallelism in each PCM bank to improve performance of DRAM-PCM hybrid memory
(Song et al., 2019). Song et al. propose data content aware PCM writes to
reduce write latency in PCM, improving performance of DRAM-PCM hybrid memory
(Song et al., 2020). Cho et al. propose Flip-N-Write to improve PCM
performance by first reading the memory content and then programming only the
bits that need to be altered (Cho and Lee, 2009). Qureshi et al. propose
PreSET, an architectural technique that SETs the PCM cells of a memory
location in the background before programming them during write (Qureshi et
al., 2012). There are also techniques to consolidate multiple write
operations, saving energy and improving performance (Xia et al., 2014) . As
MNEME addressees performance and reliability bottlenecks by tackling them at
their source, it can be combined with these and similar techniques.

###  7.2. Writeback Optimization

Several prior works propose line-level writeback, where for each evicted DRAM
cache block, processor cache blocks that become dirty are tracked and
selectively written back to PCM (Qureshi et al., 2009; Lee et al., 2010b, c,
2009b; Pourshirazi et al., 2018, 2019). Various works propose dynamic write
consolidation where PCM writes to the same row are consolidated into one write
operation (Xia et al., 2014; Stuecheli et al., 2010; Wang et al., 2013; Lee et
al., 2010d; Seshadri et al., 2014). Other works propose write activity
reduction in PCM using CPU registers (Hu et al., 2013; Huang et al., 2011).
Yet some other works propose multi-stage write operations where a write
request is served in several steps rather than in one-shot to improve
performance(Yue and Zhu, 2013; Zhang et al., 2016). Qureshi et al. propose a
morphable PCM system, which dynamically adapts between high-density and high-
latency MLC PCM and low-density and low-latency single-level cell PCM (Qureshi
et al., 2010a, b). Jiang et al. propose write truncation where a write
operation is truncated to allow read operations, compensating for data loss
using ECC (Jiang et al., 2012). MNEME is complementary to all these
approaches.

###  7.3. Page Allocation

Many modern OSes are already aware of performance and reliability
characteristics of different memory technologies (Lameter, 2013). There are
two different approaches for page placement in hybrid memory. The first
approach is to monitor the memory access patterns to pages, and migrate
access-intensive pages to the faster memory, e.g., (Yan et al., 2019; Wang et
al., 2019; Kannan et al., 2018; Chang et al., 2018; Kokolis et al., 2019;
Jayasena et al., 2018). We compare MNEME with Nimble (Yan et al., 2019) and
found it to be largely better. However, the disadvantages of this approach are
that it incurs high performance and energy overhead, as well as increasing
bank occupancy due to the movement of data in memory. An alternative approach
is to predict the memory access patterns of pages and place them in matching
memory during their initial allocation (Khouzani et al., 2015). However, it
requires accurate predictions of the memory read-write characteristics of
pages to be allocated. We not only introduce a new prediction scheme based on
first-touch instruction, but also a novel memory architecture that can be
exploited using this allocation policy. Furthermore, we discuss methods to
reduce the migration overhead inside the memory bank.

##  8\. Conclusions

We introduce MNEME, a new mechanism that enables segmented bitline
architecture in DRAM-NVM hybrid memory, introducing _intra- and inter-memory_
performance and reliability asymmetries and exploit them using an efficient
page management policy at the OS, improving both performance and reliability.
Previous architectural solutions exist to tackle page migration between
different heterogeneous units in hybrid memory. However, they lead to
significant performance and energy overhead due to high bank and channel
occupancy during data migration. In this paper, we first introduce an
architectural solution involving the use of isolation transistors in long
bitlines to create tiers with different latency and reliability
characteristics. Next, we expose the asymmetric performance and reliability
properties of memory tiers, both within and across heterogeneous memory units
of hybrid memory to the OS. The OS exploits these asymmetries in placing every
newly-referenced memory page to the best tier during its initial allocation by
predicting its data access intensity. This minimizes run-time page migrations,
which lead to performance improvements. Finally, we propose a simple approach
to facilitate page migrations within tiers of the same memory, eliminating the
need to move data over memory channels, thereby further improving performance.
We evaluate MNEME with single-core and multi-programmed workloads from the
SPEC CPU2017 Benchmark suites. Our results show that MNEME significantly
improves performance and reliability of state-of-the-art hybrid memory systems
as well as mainstream DRAM-based systems. Additionally, MNEME’s standalone
page allocation policy can also be applied to improve performance of computing
systems, where the proposed segmented bitline architecture is too costly to
incorporate.

We conclude that MNEME is a _simple yet powerful_ mechanism for hybrid memory
systems.

## References

  * jed (2019) “Non-Volatile Dual In-line Memory Module (NVDIMM) – N/F/P Specification,” _JEDEC Solid State Technology Association_ , 2019.
  * Akram et al. (2018) S. Akram, J. B. Sartor, K. S. McKinley, and L. Eeckhout, “Write-rationing garbage collection for hybrid memories,” in _PLDI_ , 2018.
  * Antognetti and Massobrio (1990) P. Antognetti and G. Massobrio, _Semiconductor device modeling with SPICE_. McGraw-Hill, Inc., 1990.
  * Apalkov et al. (2013) D. Apalkov, A. Khvalkovskiy, S. Watts, V. Nikitin, X. Tang, D. Lottis, K. Moon, X. Luo, E. Chen, A. Ong _et al._ , “Spin-transfer torque magnetic random access memory (STT-MRAM),” _JETC_ , 2013.
  * Arafa and Ramanujan (2018) M. Arafa and R. K. Ramanujan, “Memory card with volatile and non volatile memory space having multiple usage model configurations,” __US Patent 10,095,618__ , 2018.
  * Arjomand et al. (2016) M. Arjomand, M. T. Kandemir, A. Sivasubramaniam, and C. R. Das, “Boosting access parallelism to PCM-based main memory,” in _ISCA_ , 2016.
  * Balaji et al. (2019) A. Balaji, S. Song, A. Das, N. Dutt, J. Krichmar, N. Kandasamy, and F. Catthoor, “A framework to explore workload-specific performance and lifetime trade-offs in neuromorphic computing,” _CAL_ , 2019.
  * Bhattacharyya (2019) A. Bhattacharyya, “Memory arrays,” __US Patent 10,374,101__ , 2019.
  * Blagodurov et al. (2019) S. Blagodurov, G. H. Loh, and M. R. Meswani, “Hot page selection in multi-level memory hierarchies,” __US Patent 10,235,290__ , 2019.
  * Bloom (1970) B. H. Bloom, “Space/time trade-offs in hash coding with allowable errors,” _CSUR_ , 1970.
  * Bourzac (2017) K. Bourzac, “Has Intel created a universal memory technology?” _IEEE Spectrum_ , 2017.
  * Bucek et al. (2018) J. Bucek, K.-D. Lange _et al._ , “SPEC CPU2017: Next-Generation Compute Benchmark,” in _ICPE_ , 2018.
  * Cai et al. (2017) Y. Cai, S. Ghose, E. F. Haratsch, Y. Luo, and O. Mutlu, “Error characterization, mitigation, and recovery in flash-memory-based solid-state drives,” _Proceedings of the IEEE_ , 2017.
  * Cao and McAndrew (2007) Y. Cao and C. McAndrew, “MOSFET modeling for 45nm and beyond,” in _ICCAD_ , 2007.
  * Chandrasekar et al. (2012) K. Chandrasekar, C. Weis, Y. Li, B. Akesson, N. Wehn, and K. Goossens, “DRAMPower: Open-source DRAM power & energy estimation tool,” _http://www. drampower. info_ , 2012.
  * Chang et al. (2018) Y.-M. Chang, Y.-H. Chang, H.-C. Chen, and T.-W. Kuo, “Memory system and memory management method thereof,” __US Patent 10,108,555__ , 2018.
  * Cho et al. (2005) B.-H. Cho, W.-Y. Cho, H.-R. Oh, and B.-G. Choi, “Programming method of controlling the amount of write current applied to phase change memory device and write driver circuit therefor,” __US Patent 6,885,602__ , 2005.
  * Cho and Lee (2009) S. Cho and H. Lee, “Flip-N-Write: a simple deterministic technique to improve PRAM write performance, energy and endurance,” in _MICRO_ , 2009.
  * Das and Kumar (2012) A. Das and A. Kumar, “Fault-aware task re-mapping for throughput constrained multimedia applications on noc-based mpsocs,” in _RSP_ , 2012.
  * Das et al. (2013a) A. Das, A. Kumar, and B. Veeravalli, “Aging-aware hardware-software task partitioning for reliable reconfigurable multiprocessor systems,” in _CASES_ , 2013.
  * Das et al. (2013b) A. Das, A. Kumar, and B. Veeravalli, “Reliability-driven task mapping for lifetime extension of networks-on-chip based multiprocessor systems,” in _DATE_ , 2013.
  * Das et al. (2014b) A. Das, A. Kumar, and B. Veeravalli, “Communication and migration energy aware task mapping for reliable multiprocessor systems,” _FGCS_ , 2014.
  * Das et al. (2014c) A. Das, A. Kumar, and B. Veeravalli, “Energy-aware task mapping and scheduling for reliable embedded computing systems,” _TECS_ , 2014.
  * Das et al. (2014a) A. Das, A. Kumar, B. Veeravalli, C. Bolchini, and A. Miele, “Combined DVFS and mapping exploration for lifetime and soft-error susceptibility improvement in MPSoCs,” in _DATE_ , 2014.
  * Das et al. (2015) A. Das, A. Kumar, and B. Veeravalli, “Reliability and energy-aware mapping and scheduling of multimedia applications on multiprocessor systems,” _TPDS_ , 2015.
  * Das et al. (2018) A. Das, H. Hassan, and O. Mutlu, “VRL-DRAM: Improving DRAM performance via variable refresh latency,” in _DAC_ , 2018.
  * Doweck et al. (2017) J. Doweck, W.-F. Kao, A. K.-y. Lu, J. Mandelblat, A. Rahatekar, L. Rappoport, E. Rotem, A. Yasin, and A. Yoaz, “Inside 6th-generation Intel core: New microarchitecture code-named skylake,” _IEEE Micro_ , 2017.
  * Dray and Wei (2018) C. Dray and L. Wei, “High voltage tolerant word-line driver,” __US Patent 9,875,783__ , 2018.
  * Fan et al. (2014) B. Fan, D. G. Andersen, M. Kaminsky, and M. D. Mitzenmacher, “Cuckoo filter: Practically better than bloom,” in _CONEXT_ , 2014.
  * Goda et al. (2019) A. Goda, T. Vali, C. Miccoli, and P. Kalavade, “Programming memory devices,” __US Patent 10,217,515__ , 2019.
  * Hu et al. (2013) J. Hu, C. J. Xue, Q. Zhuge, W.-C. Tseng, and E. H.-M. Sha, “Write activity reduction on non-volatile main memories for embedded chip multiprocessors,” _TECS_ , 2013.
  * Huang et al. (2011) Y. Huang, T. Liu, and C. J. Xue, “Register allocation for write activity minimization on non-volatile main memory,” in _ASPDAC_ , 2011.
  * Jayasena et al. (2018) N. S. Jayasena, G. H. Loh, J. M. O’connor, and N. Chatterjee, “Page migration in a hybrid memory device,” __US Patent 9,910,605__ , 2018.
  * Jiang et al. (2012) L. Jiang, Y. Zhang, B. R. Childers, and J. Yang, “FPB: Fine-grained power budgeting to improve write throughput of multi-level cell phase change memory,” in _MICRO_ , 2012.
  * Kang et al. (2014) U. Kang, H.-s. Yu, C. Park, H. Zheng, J. Halbert, K. Bains, S. Jang, and J. S. Choi, “Co-architecting controllers and DRAM to enhance DRAM process scaling,” in _The Memory Forum_ , 2014.
  * Kannan et al. (2018) S. Kannan, A. Gavrilovska, V. Gupta, and K. Schwan, “HeteroOS: OS design for heterogeneous memory management in datacenters,” _OSR_ , 2018.
  * Khouzani et al. (2015) H. A. Khouzani, C. Yang, and J. Hu, “Improving performance and lifetime of DRAM-PCM hybrid main memory through a proactive page allocation strategy,” in _ASP-DAC_ , 2015.
  * Kim et al. (2012) Y. Kim, V. Seshadri, D. Lee, J. Liu, and O. Mutlu, “A case for exploiting subarray-level parallelism (SALP) in DRAM,” in _ISCA_ , 2012.
  * Kim et al. (2016) Y. Kim, W. Yang, and O. Mutlu, “Ramulator: A Fast and Extensible DRAM Simulator,” _CAL_ , 2016.
  * Kokolis et al. (2019) A. Kokolis, D. Skarlatos, and J. Torrellas, “PageSeer: Using page walks to trigger page swaps in hybrid memory systems,” in _HPCA_ , 2019.
  * Kültürsay et al. (2013) E. Kültürsay, M. Kandemir, A. Sivasubramaniam, and O. Mutlu, “Evaluating STT-RAM as an energy-efficient main memory alternative,” in _ISPASS_ , 2013.
  * Lameter (2013) C. Lameter, “Numa (non-uniform memory access): An overview,” _Queue_ , 2013\.
  * Lee et al. (2010b) B. C. Lee, P. Zhou, J. Yang, Y. Zhang, B. Zhao, E. Ipek, O. Mutlu, and D. Burger, “Phase-change technology and the future of main memory,” _IEEE Micro_ , 2010.
  * Lee et al. (2009a) B. C. Lee, E. Ipek, O. Mutlu, and D. Burger, “Architecting phase change memory as a scalable dram alternative,” in _ISCA_ , 2009.
  * Lee et al. (2009b) B. C. Lee, E. Ipek, O. Mutlu, and D. Burger, “Architecting phase change memory as a scalable DRAM alternative,” in _ISCA_ , 2009.
  * Lee et al. (2010c) B. C. Lee, E. Ipek, O. Mutlu, and D. Burger, “Phase change memory architecture and the quest for scalability,” _CACM_ , 2010.
  * Lee et al. (2010a) B. C. Lee, P. Zhou, J. Yang, Y. Zhang, B. Zhao, E. Ipek, O. Mutlu, and D. Burger, “Phase-change technology and the future of main memory,” _IEEE Micro_ , 2010.
  * Lee et al. (2010d) C. J. Lee, V. Narasiman, E. Ebrahimi, O. Mutlu, and Y. N. Patt, “DRAM-aware last-level cache writeback: Reducing write-caused interference in memory systems,” 2010.
  * Lee et al. (2013) D. Lee, Y. Kim, V. Seshadri, J. Liu, L. Subramanian, and O. Mutlu, “Tiered-latency DRAM: A low latency and low cost DRAM architecture,” in _HPCA_ , 2013.
  * Luk et al. (2005) C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood, “Pin: Building customized program analysis tools with dynamic instrumentation,” in _PLDI_ , 2005.
  * Lung et al. (2016) H.-L. Lung, C. P. Miller, C.-J. Chen, S. C. Lewis, J. Morrish, T. Perri, R. C. Jordan, H.-Y. Ho, T.-S. Chen, W.-C. Chien _et al._ , “A double-data-rate 2 (DDR2) interface phase-change memory with 533MB/s read-write data rate and 37.5 ns access latency for memory-type storage class memory applications,” in _IMW_ , 2016.
  * Mallik et al. (2017) A. Mallik, D. Garbin, A. Fantini, D. Rodopoulos, R. Degraeve, J. Stuijt, A. Das, S. Schaafsma, P. Debacker, G. Donadio _et al._ , “Design-technology co-optimization for OxRRAM-based synaptic processing unit,” in _VLSI Technology_ , 2017.
  * Mutlu (2013) O. Mutlu, “Memory scaling: A systems architecture perspective,” in _IMW_ , 2013.
  * Mutlu (2017) O. Mutlu, “The RowHammer problem and other issues we may face as memory becomes denser,” in _DATE_ , 2017.
  * Mutlu and Kim (2019) O. Mutlu and J. S. Kim, “Rowhammer: A retrospective,” _TCAD_ , 2019.
  * Mutlu and Subramanian (2015) O. Mutlu and L. Subramanian, “Research problems and opportunities in memory systems,” _Supercomputing Frontiers and Innovations_ , 2015.
  * Nirschl et al. (2007) T. Nirschl, J. Philipp, T. Happ, G. W. Burr, B. Rajendran, M.-H. Lee, A. Schrott, M. Yang, M. Breitwisch, C.-F. Chen _et al._ , “Write strategies for 2 and 4-bit multi-level phase-change memory,” in _IEDM_ , 2007\.
  * Poremba et al. (2015) M. Poremba, T. Zhang, and Y. Xie, “Nvmain 2.0: A user-friendly memory simulator to model (non-) volatile memory systems,” _CAL_ , 2015.
  * Pourshirazi et al. (2018) B. Pourshirazi, M. V. Beigi, Z. Zhu, and G. Memik, “WALL: A writeback-aware LLC management for PCM-based main memory systems,” in _DATE_ , 2018.
  * Pourshirazi et al. (2019) B. Pourshirazi, M. V. Beigi, Z. Zhu, and G. Memik, “Writeback-aware LLC management for PCM-based main memory systems,” _TODAES_ , 2019.
  * Qureshi et al. (2009) M. K. Qureshi, V. Srinivasan, and J. A. Rivers, “Scalable high performance main memory system using phase-change memory technology,” in _ISCA_ , 2009\.
  * Qureshi et al. (2010a) M. K. Qureshi, M. M. Franceschini, L. A. Lastras-Montaño, and J. P. Karidis, “Morphable memory system: A robust architecture for exploiting multi-level phase change memories,” in _ISCA_ , 2010.
  * Qureshi et al. (2010b) M. K. Qureshi, M. M. Franceschini, and L. A. Lastras-Montano, “Improving read performance of phase change memories via write cancellation and write pausing,” in _HPCA_ , 2010.
  * Qureshi et al. (2012) M. K. Qureshi, M. M. Franceschini, A. Jagmohan, and L. A. Lastras, “PreSET: improving performance of phase change memories by exploiting asymmetry in write times,” in _ISCA_ , 2012.
  * Redaelli (2018) A. Redaelli, “Phase Change Memory: Device Physics, Reliability and Applications,” _Phase Change Memory_ , 2018.
  * Redaelli and Perrone (2017) A. Redaelli and C. Perrone, “Semiconductor constructions and memory arrays,” __US Patent 9,748,480__ , 2017.
  * Ren et al. (2015) J. Ren, J. Zhao, S. Khan, J. Choi, Y. Wu, and O. Mutlu, “ThyNVM: Enabling software-transparent crash consistency in persistent memory systems,” in _MICRO_ , 2015.
  * Rixner et al. (2000) S. Rixner, W. J. Dally, U. J. Kapasi, P. Mattson, and J. D. Owens, “Memory access scheduling,” in _ISCA_ , 2000.
  * Sadasivam et al. (2017) S. K. Sadasivam, B. W. Thompto, R. Kalla, and W. J. Starke, “IBM Power9 processor architecture,” _IEEE Micro_ , 2017.
  * Sandhu et al. (2020) B. S. Sandhu, C. Pietrzyk, and G. M. Lattimore, “Memory write driver, method and system,” __US Patent 10,529,420__ , 2020.
  * Seong et al. (2010) N. H. Seong, D. H. Woo, and H.-H. S. Lee, “Security Refresh: Prevent malicious wear-out and increase durability for phase-change memory with dynamically randomized optaddress mapping,” in _ISCA_ , 2010.
  * Seshadri et al. (2013) V. Seshadri, Y. Kim, C. Fallin, D. Lee, R. Ausavarungnirun, G. Pekhimenko, Y. Luo, O. Mutlu, P. B. Gibbons, M. A. Kozuch _et al._ , “RowClone: fast and energy-efficient in-DRAM bulk data copy and initialization,” in _MICRO_ , 2013.
  * Seshadri et al. (2014) V. Seshadri, A. Bhowmick, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “The dirty-block index,” in _ISCA_ , 2014.
  * Song et al. (2019) S. Song, A. Das, O. Mutlu, and N. Kandasamy, “Enabling and exploiting partition-level parallelism (PALP) in phase change memories,” _TECS_ , 2019\.
  * Song et al. (2020) S. Song, A. Das, O. Mutlu, and N. Kandasamy, “Improving phase change memory performance with data content aware access,” in _ISMM_ , 2020.
  * Srinivasan et al. (2004) J. Srinivasan, S. V. Adve, P. Bose, and J. A. Rivers, “The case for lifetime reliability-aware microprocessors,” in _ISCA_ , 2004.
  * Stine et al. (2007) J. E. Stine, I. Castellanos, M. Wood, J. Henson, F. Love, W. R. Davis, P. D. Franzon, M. Bucher, S. Basavarajaiah, J. Oh _et al._ , “FreePDK: An open-source variation-aware design kit,” in _MSE_ , 2007.
  * Stuecheli et al. (2010) J. Stuecheli, D. Kaseridis, D. Daly, H. C. Hunter, and L. K. John, “The virtual write queue: Coordinating dram and last-level cache policies,” in _ISCA_ , 2010.
  * Villa (2018) C. Villa, “PCM array architecture and management,” in _Phase Change Memory_ , 2018.
  * Wang et al. (2019) X. Wang, H. Liu, X. Liao, J. Chen, H. Jin, Y. Zhang, L. Zheng, B. He, and S. Jiang, “Supporting superpages and lightweight page migration in hybrid memory systems,” _TACO_ , 2019.
  * Wang et al. (2013) Z. Wang, S. Shan, T. Cao, J. Gu, Y. Xu, S. Mu, Y. Xie, and D. A. Jiménez, “WADE: Writeback-aware dynamic cache management for NVM-based main memory system,” _TACO_ , 2013.
  * Wilkes (2001) M. V. Wilkes, “The memory gap and the future of high performance memories,” _Computer Architecture News_ , 2001.
  * Wong et al. (2010) H.-S. P. Wong, S. Raoux, S. Kim, J. Liang, J. P. Reifenberg, B. Rajendran, M. Asheghi, and K. E. Goodson, “Phase change memory,” _Proceedings of the IEEE_ , 2010.
  * Wulf and McKee (1995) W. A. Wulf and S. A. McKee, “Hitting the memory wall: implications of the obvious,” _Computer Architecture News_ , 1995.
  * Xia et al. (2014) F. Xia, D. Jiang, J. Xiong, M. Chen, L. Zhang, and N. Sun, “DWC: Dynamic write consolidation for phase change memory systems,” in _ICS_ , 2014.
  * Yan et al. (2019) Z. Yan, D. Lustig, D. Nellans, and A. Bhattacharjee, “Nimble page management for tiered memory systems,” in _ASPLOS_ , 2019.
  * Yoon et al. (2015) H. Yoon, J. Meza, N. Muralimanohar, N. P. Jouppi, and O. Mutlu, “Efficient data mapping and buffering techniques for multilevel cell phase-change memories,” _TACO_ , 2015.
  * Yue and Zhu (2013) J. Yue and Y. Zhu, “Accelerating write by exploiting pcm asymmetries,” in _HPCA_ , 2013.
  * Zhang et al. (2016) L. Zhang, B. Neely, D. Franklin, D. Strukov, Y. Xie, and F. T. Chong, “Mellow writes: Extending lifetime in resistive memories through selective slow write backs,” in _ISCA_ , 2016.


# Computational Models for High-Power Cyclotrons and FFAs

Andreas Adelmann ,11footnotetext: Corresponding author. 0000-0002-7230-7007
Chris T. Rogers

###### Abstract

A summary of numerical modeling capabilities regarding high power cyclotrons
and fixed field alternating gradient machines is presented. This paper focuses
on techniques made available by the OPAL simulation code.

##  1 Overview on Computational Models

In all high-power particle accelerators "one of the major limitations is
particle losses. Losses may be controlled, resulting in beam particles
impinging on dedicated equipment such as collimators, or uncontrolled,
resulting in beam particles striking other equipment around the accelerator.
Uncontrolled losses can damage and activate any equipment in the accelerator
and so must be minimized. Controlled losses need to be carefully considered
and also minimized. The amount and cause of loss are investigated by modeling
accelerators using simulation codes that model numerically the behaviour of
beams. A review of available numerical codes can be found in the article of
Smirnov [1]. In this paper modeling capabilities available in OPAL are
discussed in more detail [2].

###  1.1 Single particle modeling

For conventional cyclotrons (and FFAs) the single particle tool box is
established and many different codes variants exists [1]. For cyclotrons and
(horizontal FFAs) the existing tools seem to be comfortable and accurate. New
machines like vertical FFAs, currently studied for example at the Rutherford
Appleton Laboratory (RAL) [3], require non–trivial modifications to the
existing codes. These modifications are on the way for example in the code
OPAL [2] and expected to be available in second quarter of 2022.

Recently, in the context of very high field and ultra compact H- cyclotrons
beam stripping losses of ion beams by interactions with residual gas and
electromagnetic fields are evaluated [4]. The beam stripping algorithm,
implemented in OPAL, evaluates the interaction of hydrogen ions with residual
gas and electromagnetic fields. In the first case, the cross sections of the
processes are estimated according to the energy by means of analytical
functions (see Sec. II-A c[4]). The implementation allows the user to set the
pressure, temperature, and composition of the residual gas, which could be
selected for the calculations as either molecular hydrogen
($\mathrm{H}_{2}^{+}$) or dry air in the usual proportion. For precise
simulations, a two-dimensional pressure field map from an external file can be
imported into OPAL, providing more realistic vacuum conditions.

Concerning electromagnetic stripping, the electric dissociation lifetime is
evaluated through the theoretical formalism (see Sec. II-B [4]). In both
instances, the individual probability at each integration step for every
particle is assessed.

A stochastic process is used to evaluate if an interaction occurs. In this
case the particle will be stripped and removed from the beam, or optionally
transformed to a secondary heavy particle, dependent on the interaction. In
this case, the secondary particle will continue its movement but with the new
particle properties.

###  1.2 Large Scale Multiparticle Modeling

In general, modeling losses in high intensity accelerators require 3D space-
charge and sufficient simulation particles. Recent investigations [5] propose
a sparse grid-based adaptive noise reduction strategy for electrostatic
particle-in-cell (PIC) simulations. By projecting the charge density onto
sparse grids, high-frequency particle noise is reduced and hence an optimal
number of grid points and simulation particles can be obtained. For a 3D
Penning trap simulation, a maximum speedup of 2.8 and 15 times memory
reduction has been obtained. This method is already integrated into OPAL.

###  1.3 Surrogate Model Construction

Cheap to evaluate surrogate models have gained a lot of interest lately.
Statistical [6] or machine learning techniques are used [7]. These models can
for example replace a computationally heavy model in a multi-objective
optimization [8] or in the future be part of an on-line model. Some surrogate
modeling algorithms may include an intrinsic estimator for the model
uncertainty [9].

##  2 Physics Modeling

In this section we show latest additions to the open source code OPAL [2]
regarding cyclotron and FFA modeling capabilities.

###  2.1 Modeling H- Injection and Painting in Vertical and Horizontal FFAs

Fixed Field Accelerators (FFAs) have fixed magnetic fields, like cyclotrons,
but increase bending field with momentum and hence more compact designs can be
realized. FFAs offer the power efficiency of cyclotrons combined with the
energy reach of synchrotrons.

FFAs have never been used for high power proton acceleration, however in OPAL
the necessary models are available for design. Single particle tracking has
been benchmarked against the KURNS FFA [10]. A design for a 3-12 MeV H- FFA
prototype ring is being pursued at RAL as a prototype for a MW-class neutron
spallation source [3]. Scaling horizontal orbit excursion (hFFA) and a
vertical orbit excursion (vFFA) FFA are both under consideration. Both are
non–isochronous machines using RF cavities with variable resonant frequency.
Injection is planned using charge exchange of H- to H+ and phase space
painting.

In hFFAs, magnetic rigidity varies with radius. The dipole field varies as
[11]

$B_{z}(z=0)=B_{0}(\psi)\left(\frac{r}{r_{0}}\right)^{k}.$ (2.1)

$B_{0}(\psi)$ is the dipole field as a function of a normalised azimuthal
coordinate $\psi$, $r$ is the radial coordinate, $r_{0}$ is a nominal (user-
defined) radius, and $k$ is the field index. The field away from the midplane,
at $z\neq 0$, may be calculated using a recursion relation arising from
consideration of Maxwell’s equations in free space. OPAL has capability to
calculate the expansion to arbitrary order, within machine precision. The
normalised azimuthal coordinate

$\psi=\phi-\tan(\delta)\ln\left(\frac{r}{r_{0}}\right)$ (2.2)

is a measure of distance around the ring. Here $\phi$ is the geometrical
azimuthal angle and $\delta$ is the spiral angle; for a sector FFA magnet
$\delta=0$ and $\psi=\phi$. The arrangement of fields in this way guarantees
that single particle trajectories and optical parameters at all orders scale
exactly with momentum.

In vFFAs, magnetic rigidity varies with height. As particles are accelerated,
the closed orbit changes height. Successive acceleration kicks add
incoherently, so overall the beam follows the closed orbit with no appreciable
emittance growth. Rectangular vFFA magnets have been implemented in OPAL, with
a dipole field that varies as [12]

$B_{0}(x_{v}=0)=B_{0}(s_{v})e^{mz_{v}}.$ (2.3)

$z_{v}$ is the height, $s_{v}$ is a nominal longitudinal coordinate and
$x_{v}$ is a nominal horizontal coordinate in the rectangular coordinate
system of the magnet. $B_{0}$ describes the dipole field variation with
longitudinal distance. A $\tanh$ model is available for vFFA fields. $m$ is
the vFFA field index, roughly equivalent to the field index $k$ in hFFAs.
Fields away from the plane having $x_{v}=0$ are calculated using a field
expansion derived from consideration of Maxwell’s laws. It is noted that the
focusing in the magnet body is, to linear order, skew quadrupole. The fringe
field has solenoid components parallel to $s_{v}$ that may be significant for
short magnets. This arrangement of fields guarantees that trajectories and
optical functions are identical as momentum increases, barring a vertical
displacement. In particular, the path length of the beam is independent of
momentum, the momentum compaction factor is exactly 0 and ultra-relativistic
particles are isochronous.

In order to model injection into the FFA, OPAL was extended with models for:

  * •

horizontal & vertical FFA magnets as described above;

  * •

variable frequency RF cavities;

  * •

arbitrary order multipoles with maxwellian fringe fields;

  * •

foil model (scattering and energy loss);

  * •

pulsed injected beam; and

  * •

pulsed multipoles.

All but the latter two features are available in the latest version of OPAL.
This enabled a fully four-dimensional simulation of the injection system,
including consideration of effects such as appropriate phasing of the pulsed
dipoles and transverse breathing of the beam arising due to initial
longitudinal mismatch at injection.

As an example, a schematic of an injection system and associated parameters
for the 3-12 MeV test ring is shown for a horizontal FFA in Fig. 1. Owing to
the compact nature of the ring, the injection system is spread across a number
of cells. H- are brought into the ring and onto a foil. Bump magnets in the
ring distort the proton closed orbit so that particles passing through the
foil are returned to a nominal closed orbit. The foil is placed inside the
defocusing (D) dipole magnet so that the distorted H+ closed orbit and H-
beam, initially separated, are brought onto the same trajectory. Electrons are
stripped from the H- leaving H+ (protons). The bump magnets are slowly varied,
so that the proton closed orbit is moved away from the injection point for the
H- and newly injected particles are at higher horizontal amplitude. In the H-
injection line, pulsed magnets move the H- upwards so that newly injected
particles are at higher vertical amplitude. Overall, a correlation is
introduced between horizontal and vertical amplitude. Sample trajectories and
bump magnet field strengths for the magnets in the ring are shown in Fig. 1.
In this example vertical bumpers are not considered - they are all kept at 0 T
field. The beam following injection is shown in fig. 2.

Figure 1: Injection system for the hFFA (Left) field map of the hFFA,
calculated using OPAL, with labels indicating the position of injection
equipment (top right) closed orbits for different bump magnets (bottom right)
required bump magnet fields.

Figure 2: Beam (left) after injection is completed, but still on a distorted
orbit (right) following collapse of the bump. $x$ is the position of the beam
relative to the ring centre and $y$ is the height of the particle above the
midplane. Particles are coloured according to the injection turn.

###  2.2 Beam stripping interactions

Beam transmission optimization and loss characterization, where beam stripping
interactions are a key issue, play an important role in the design and
operation of compact cyclotrons. A beam stripping model has been implemented
in the three-dimensional object-oriented parallel code OPAL-cycl, a flavor of
the OPAL framework. The model includes Monte Carlo methods for interaction
with residual gas and dissociation by electromagnetic stripping. The model has
been verified with theoretical models and it has been applied to the AMIT
cyclotron according to design conditions [4].

###  2.3 Spiral inflector modeling

In [13] a spiral inflector model implemented in OPAL is presented, that
enables us to run highly realistic simulations of the spiral inflector system
of a compact cyclotron (c.f. Fig. 3). A new geometry class and field solver
can handle the complicated boundary conditions posed by the electrode system
in the central region of the cyclotron both in terms of particle termination,
and calculation of self-fields. Results are benchmarked against the analytical
solution of a coasting beam. As a practical example, the spiral inflector and
the first revolution in a $1$ MeV/amu test cyclotron, located at Best
Cyclotron Systems, Inc., are modeled and compared to the simulation results
[14, 15]. In conclusion, OPAL can handle realistic and arbitrary boundary
geometries. Simulated injection efficiencies and beam shape compare well with
measured efficiencies and a preliminary measurement of the beam distribution
after injection.

Figure 3: Spiral inflector with selected particle trajectories from an OPAL
simulation. The beam enters axially (from the top) through an aperture (grey)
and is bent into the mid-plane by a combination of the electrostatic field
generated by the spiral electrodes (green and blue) and the cyclotron’s main
magnetic field. Then it is accelerated by the two Dees (copper, Dummy-Dees not
shown) [13].

###  2.4 Neighboring Turn Modeling

This article presents a hardware architecture independent implementation of an
adaptive mesh refinement Poisson solver that is integrated into the
electrostatic Particle-In-Cell beam dynamics code OPAL. The Poisson solver is
solely based on second generation Trilinos packages to ensure the desired
hardware portability. Based on the massively parallel framework AMREX,
formerly known as BoxLib, the new adaptive mesh refinement interface provides
several refinement policies in order to enable precise large-scale
neighbouring bunch simulations in high intensity cyclotrons. The solver is
validated with a built-in multigrid solver of AMREX and a test problem with
analytical solution. The parallel scalability is presented as well as an
example of a neighbouring bunch simulation that covers the scale of the later
anticipated physics simulation [16].

Figure 4:  Integrated projection of the electric field component $E_{x}$ onto
the xy-plane showing 7 adjacent particle bunches [16].

##  3 Path Forward

While statistical and machine learning techniques have a lot of potential,
high fidelity physics simulations will always be used to, for example, produce
the training set. In case of high-intensity machines we will need large
numbers of particles and the associated fine mesh to solve the PDE in
question. It is imperative that we make use of existing and future high
performance infrastructure. A performance portable implementation [16] is of
utmost importance. The OPAL collaboration [2] is in the progress to completely
rewrite the code according to the sketch in Fig. 5. With this new architecture
we will be able to make efficient use of Exascale-Architecture that will come
online soon.

Figure 5:  Outlook of the future OPAL architecture, targeting in a performance
portable way future exascale architectures.

The core algorithms of OPAL are already performance portable as demonstrated
in [17].

## Acknowledgments

The authors acknowledge the OPAL developer team for their continued support of
this open source, community-driven code.

## References

  * Smirnov [2017] V. Smirnov.  Computer codes for beam dynamics analysis of cyclotronlike accelerators.  _Phys. Rev. Accel. Beams_ , 20:124801, 12 2017.  doi: 10.1103/PhysRevAccelBeams.20.124801.  URL https://link.aps.org/doi/10.1103/PhysRevAccelBeams.20.124801.
  * OPA [2021] The OPAL Framework: Version 2.4, 2021.  http://amas.web.psi.ch/opal/Documentation/2.4/index.html.
  * Machida et al. [2021] S. Machida, D. J. Kelliher, J-B. Lagrange, and C. T. Rogers.  Optics design of vertical excursion fixed-field alternating gradient accelerators.  _Phys. Rev. Accel. Beams_ , 24:021601, 2 2021.  doi: 10.1103/PhysRevAccelBeams.24.021601.  URL https://link.aps.org/doi/10.1103/PhysRevAccelBeams.24.021601.
  * Calvo et al. [2021] P. Calvo, I. Podadera, D. Gavela, C. Oliver, A. Adelmann, J. Snuverink, and A. Gsell.  Beam stripping interactions in compact cyclotrons.  _Phys. Rev. Accel. Beams_ , 24:090101, 11 2021.  doi: 10.1103/PhysRevAccelBeams.24.090101.  URL https://link.aps.org/doi/10.1103/PhysRevAccelBeams.24.090101.
  * Muralikrishnan et al. [2021] Sriramkrishnan Muralikrishnan, Antoine J. Cerfon, Matthias Frey, Lee F. Ricketson, and Andreas Adelmann.  Sparse grid-based adaptive noise reduction strategy for particle-in-cell schemes.  _Journal of Computational Physics: X_ , 11:100094, 2021\.  ISSN 2590-0552.  doi: https://doi.org/10.1016/j.jcpx.2021.100094.  URL https://www.sciencedirect.com/science/article/pii/S2590055221000111.
  * Adelmann [2019] Andreas Adelmann.  On nonintrusive uncertainty quantification and surrogate model construction in particle accelerator modeling.  _SIAM/ASA Journal on Uncertainty Quantification_ , 7(2):383–416, 2019.
  * Bellotti et al. [2021] Renato Bellotti, Romana Boiger, and Andreas Adelmann.  Fast, efficient and flexible particle accelerator optimisation using densely connected and invertible neural networks.  _Information_ , 12(9), 2021.  doi: 10.3390/info12090351.  URL https://www.mdpi.com/2078-2489/12/9/351.
  * Edelen et al. [2020] Auralee Edelen, Nicole Neveu, Yannick Huber, Matthias Frey, and Andreas Adelmannn.  Machine learning to enable orders of magnitude speedup in multi-objective optimization of particle accelerator systems’.  _Phys. Rev. AB_ , 23:044601, 2020.  doi: 10.1103/PhysRevAccelBeams.23.044601.  URL https://link.aps.org/doi/10.1103/PhysRevAccelBeams.23.044601.
  * Frey and Adelmann [2021] Matthias Frey and Andreas Adelmann.  Global sensitivity analysis on numerical solver parameters of particle-in-cell models in particle accelerator systems.  _Computer Physics Communications_ , 258:107577, 2021.  ISSN 0010-4655.  doi: https://doi.org/10.1016/j.cpc.2020.107577.  URL http://www.sciencedirect.com/science/article/pii/S0010465520302770.
  * Sheehy et al. [2015] Suzanne Sheehy et al.  Progress on Simulation of Fixed Field Alternating Gradient Accelerators.  In _6th International Particle Accelerator Conference_ , page MOPJE077, 2015.  doi: 10.18429/JACoW-IPAC2015-MOPJE077.
  * Symon et al. [1956] K. R. Symon, D. W. Kerst, L. W. Jones, L. J. Laslett, and K. M. Terwilliger.  Fixed-field alternating-gradient particle accelerators.  _Phys. Rev._ , 103:1837–1859, Sep 1956.  doi: 10.1103/PhysRev.103.1837.  URL https://link.aps.org/doi/10.1103/PhysRev.103.1837.
  * Brooks [2013] Stephen Brooks.  Vertical orbit excursion fixed field alternating gradient accelerators.  _Phys. Rev. ST Accel. Beams_ , 16:084001, Aug 2013.  doi: 10.1103/PhysRevSTAB.16.084001.  URL https://link.aps.org/doi/10.1103/PhysRevSTAB.16.084001.
  * Winklehner et al. [2017a] Daniel Winklehner, Andreas Adelmann, Achim Gsell, Tulin Kaman, and Daniela Campo.  Realistic simulations of a cyclotron spiral inflector within a particle-in-cell framework.  _Physical Review Accelerators and Beams_ , 20(12):124201, 12 2017a.  doi: 10.1103/PhysRevAccelBeams.20.124201.  URL https://link.aps.org/doi/10.1103/PhysRevAccelBeams.20.124201.
  * Winklehner et al. [2017b] Daniel Winklehner, Andreas Adelmann, Achim Gsell, Tulin Kaman, and Daniela Campo.  Realistic simulations of a cyclotron spiral inflector within a particle-in-cell framework.  _Phys. Rev. Accel. Beams_ , 20:124201, Dec 2017b.  doi: 10.1103/PhysRevAccelBeams.20.124201.  URL https://link.aps.org/doi/10.1103/PhysRevAccelBeams.20.124201.
  * Alonso et al. [2015] J. Alonso, S. Axani, L. Calabretta, D. Campo, L. Celona, J.M. Conrad, A. Day, G. Castro, F. Labrecque, and D. Winklehner.  The isodar high intensity h2+ transport and injection tests.  _Journal of Instrumentation_ , 10(10):T10003, oct 2015.  doi: 10.1088/1748-0221/10/10/T10003.  URL https://dx.doi.org/10.1088/1748-0221/10/10/T10003.
  * Frey et al. [2021] Matthias Frey, Andreas Adelmann, and Uldis Locans.  On architecture and performance of adaptive mesh refinement in an electrostatics particle-in-cell code (vol 247, 106912, 2020).  _COMPUTER PHYSICS COMMUNICATIONS_ , 265, 2021.
  * Muralikrishnan et al. [2022] Sriramkrishnan Muralikrishnan, Matthias Frey, Alessandro Vinciguerra, Michael Ligotino, Antoine J. Cerfon, Miroslav Stoyanov, Rahulkumar Gayatri, and Andreas Adelmann.  Alpine: A set of performance portable plasma physics particle-in-cell mini-apps for exascale computing, 2022.  URL arXiv:2205.11052.