and 5.5% on average) over DDR4 mainly due to reduction in execution time.

Second, reduces DRAM energy by 6.8%/6.7% on average for single-core and
4c-high workloads by especially eliminating some unnecessary refresh
operations. The maximum DRAM energy reduction that can be achieved by
completely eliminating DRAM refresh operation is 7.8% as indicated by
“NoRefresh” DRAM.

Third, -Combined reduce DRAM energy by 6.1% for both single-core and 4c-high
workloads. This shows that the energy overhead of and are small.

The new -based refresh mechanisms eliminate most of the energy overhead of
DRAM refresh. -based RowHammer protection and memory scrubbing improve DRAM
reliability and security with low DRAM energy cost.

####  5.6.4 Sensitivity Studies

We analyze the performance and DRAM energy effects of different configuration
parameters.

#####  5.6.4.1 DRAM Refresh Period

Fig. 5.11 plots the speedup and energy reduction that , , and -combined
achieve over DDR4 for different refresh periods across single- and four-core
workloads. We make three key observations. First, the performance and energy
benefits of / increase as the refresh period reduces, achieving 50.5%/53.6%
speedup and 25.6%/29.0% DRAM energy reduction on average at
$8\text{\,}\mathrm{ms}$ refresh period across 4c-high workloads. Thus, -based
refresh mechanisms will become even more beneficial to employ in future DRAM
chips that are expected to require more frequent refresh [43, 42, 65]. Second,
-combined provides 51.5% speedup and 27.9% DRAM energy reduction on average
across 4c-high at $8\text{\,}\mathrm{ms}$ refresh period, showing that the
overheads of -based RowHammer protection and memory scrubbing mechanisms
remain low even at high refresh rates. Third, all -based maintenance mechanism
eliminate most of the performance overhead of refresh across all refresh
periods (e.g., -combined reaches 95.6% of the hypothetical _NoRefresh_ DRAM at
$8\text{\,}\mathrm{ms}$).

?figurename? 5.11: Sensitivity to Refresh Period

We conclude that -based maintenance mechanisms improve DRAM reliability and
security while greatly reducing the overhead of DDR4 refresh. Their
performance benefits are expected to increase for future DRAM chips that are
likely to require high refresh rates.

#####  5.6.4.2 Lock Region Size

The size of a lock region affects the performance and energy of -based
maintenance mechanisms. Generally, it is desirable to have many small-sized
lock regions to enable locking smaller portions of the DRAM chip for
maintenance and allow accesses to non-locked regions. However, the DRAM chip
area overhead of increases as the number of lock regions in a bank increase.
We analyze the performance and energy impact of different lock region size
configurations.

We use the following equation to calculate the number of lock regions per bank
($Num_{LRs/bank}$) by dividing the number of rows in a bank
($Num_{rows/bank}$) to the number of rows in a lock region ($Num_{rows/LR}$).

$Num_{LRs/bank}=\frac{Num_{rows/bank}}{Num_{rows/LR}}$ (5.3)

Figure 5.12 shows the speedup and DRAM energy savings that , , and -Combined
achieve for different number of lock regions per bank across 4c-high
workloads. We make two key observations. First, with a single lock region per
bank, /-Combined on average perform 3.7%/8.9% worse than the DDR4 baseline,
which leads to 3.2%/10.7% higher average DRAM energy consumption. This is
because a maintenance operation locks the entire bank, and thus the MC waits
for upon receiving an to issue another ACT to the same bank. Although suffers
from the same issue, it still outperforms the baseline by eliminating
unnecessary refreshes to rows with high retention times.

?figurename? 5.12: Sensitivity to Lock Region Size.

Second, when the lock regions are smaller, and thus a bank has four or more
lock regions, all mechanisms outperform the baseline for all 4c-high workloads
and save DRAM energy. This shows that even a small number of lock regions are
sufficient to overlap the maintenance operation latency with accesses to non-
locked regions.

Third, the performance and energy benefits of increasing the number of lock
regions gradually diminish. We find that $16$ 8192-row lock regions incur low
area overhead and achieve a significant portion of the speedup that smaller
lock regions provide (e.g., -Combined achieves 7.7%/9.3% speedup with 16/256
lock regions). Therefore, by default, we use $16$ lock regions per bank in our
evaluations.

We conclude that significantly reduces the overhead of DRAM maintenance
operations even with large lock regions.

#####  5.6.4.3  Divergence Across Chips

In Section 5.3.2, we explain divergence in maintenance operations can happen
when different DRAM chips in the same rank perform maintenance operation at
different times. Such a divergence leads to a partial row activation when the
activated row is in a locked region in some DRAM chips but not in others. To
handle partial row activations, we develop three policies.

Precharge. With the _Precharge_ policy, the MC issues a PRE command to close
the partially activated row when some DRAM chips send but others do not. After
closing the partially activated row, the MC can attempt to activate a row from
a different lock region in the same bank.

Wait. With the _Wait_ policy, the MC issues multiple ACT commands until a
partially activated row becomes fully activated. When some chips send for a
particular ACT but others do not, the MC waits for and issues a new ACT to
attempt activating the same row in DRAM chips that previously sent .

Hybrid. We also design a _Hybrid_ policy, where the memory controller uses the
_Precharge_ policy to close a partially activated row if the request queue
contains $N$ or more requests that need to access rows in different lock
regions in the same bank. If the requests queue has less than $N$ requests to
different lock regions, the MC uses the _Wait_ policy to retry activating the
rest of the partially activated row.

In Fig. 5.13, we compare the performance and energy savings of and when using
the three divergence handling policies across 4c-high workloads. The plots
show results for the common-case (CC) and worst-case (WC) scenarios with
regard to when maintenance operations happen across different chips in the
same rank. In the common-case scenario, the DRAM chips generally refresh the
same row at the same time due to sharing the same DRAM architecture design.
However, refresh operations in some of the DRAM chips may still diverge during
operation depending on the refresh mechanism that is in use. For example,
refreshes retention-weak rows, whose locations may differ across the DRAM
chips, at a higher rate compared to other rows, resulting in divergence in
refresh operations across the DRAM chips in a rank. In the worst-case
scenario, we deliberately configure the DRAM chips to refresh different rows
at different times. For this, we 1) delay the first refresh operation in
$chip_{i}$ by $i\times l{ref}$, where $0\leq i<Num{chips/rank}$ and $l_{ref}$
is the latency of a single refresh operation, and 2) set the _Lock Region
Counter (LRC)_ of $chip_{i}$ to $i$.

?figurename? 5.13: Comparison of different policies for handling refresh
divergence across DRAM chips.

We make three key observations. First, the performance and DRAM energy
consumption only slightly varies across different divergence handling policies
in both common-case and worst-case scenarios. This is because, after a partial
row activation happens, a different row that is not in a locked region in all
off of the DRAM chips often does not exist in the memory request queue. As a
result, the _Precharge_ and _Hybrid_ policies perform similarly to the _Wait_
policy.

Second, performs worse than the DDR4 baseline in the worst-case refresh
distribution scenario (i.e., average slowdown of 2.6% with the _Wait_ policy).
Certain individual workloads experience even higher slowdown (up to 12.8% with
the _Wait_ policy). The reason is that, when $N$ different DRAM chips lock a
region at different times, the total duration during which the lock region is
unavailable becomes $N$ times the duration when all chips simultaneously
refresh the same lock region. This significantly increases the performance
overhead of refresh operations.

Third, does not suffer much from the divergence problem and outperforms the
DDR4 baseline even in the worst-case scenario. This is because mitigates the
DRAM refresh overhead by significantly reducing the number of total refresh
operations by exploiting retention-time variation across DRAM rows. Thus, the
benefits of eliminating many unnecessary refresh operations surpass the
overhead of divergence.

We conclude that 1) although suffers from noticeable slowdown for the worst-
case scenario, which should not occur in a well-designed system, it still
provides comparable performance to conventional DDR4 and 2) outperforms the
baseline and saves DRAM energy even in the worst-case scenario.

#####  5.6.4.4 Comparison to Conventional DRAM Scrubbing

Fig. 5.14 compares the performance and energy overheads of conventional DDR4
scrubbing to across 4c-high workloads. In the figure, _DDR4 Scrubbing_
represents the performance and energy overhead of conventional scrubbing
compared to DDR4 without memory scrubbing. Similarly, represents the
performance and DRAM energy overhead of -based scrubbing compared to .

?figurename? 5.14: DDR4 Scrubbing vs. .

We make two observations. First, both DDR4 scrubbing and have negligible
performance overhead for scrubbing periods of 5 minutes and larger because
scrubbing operations are infrequent at such periods. Second, DDR4 scrubbing
causes up to 1.49%/8.84% average slowdown for 1 minute/10 second scrubbing
period (up to 2.37%/14.26%), while causes only up to 0.34%/1.78% slowdown.
DDR4 scrubbing has high overhead at low scrubbing periods because moving data
from DRAM to the MC to perform scrubbing is inefficient compared to performing
scrubbing within DRAM using . Scrubbing at high rates may become necessary for
future DRAM chips as their reliability characteristics continuously worsen.
Additionally, mechanisms that improve DRAM performance at the cost of reduced
reliability [49, 221] can use frequent DRAM scrubbing to achieve the desired
DRAM reliability level.

We conclude that SMS performs memory scrubbing more efficiently than
conventional MC based scrubbing and it enables scrubbing at high rates with
small performance and energy overheads.

#####  5.6.4.5 Comparison to PARA in Memory Controller

Fig. 5.15 compares the performance and energy overheads of PARA implemented in
the MC (as proposed by Kim et al. [73]) for DDR4 and for different neighbor
row activation probabilities (i.e., $P_{mark}$) across 4c-high workloads.
_PARA_ represents the performance and energy overheads with respect to
conventional DDR4 with no RowHammer protection. Similarly, represents the
performance and energy overheads with respect to a chip, which uses for
periodic refresh, with no RowHammer protection.

?figurename? 5.15: PARA vs. .

We make two observations. First, the performance and energy consumption of MC-
based PARA scales poorly with $P_{mark}$. At the default $P_{mark}$ of 1%,
PARA incurs 12.4%/11.5% average performance/DRAM energy overhead. For higher
$P_{mark}$, the overheads of PARA increase dramatically to 81.4%/368.1% at
$P_{mark}$ of 20%. Second, is significantly more efficient than PARA. At the
default $P_{mark}$ of 1%, incurs only 1.4%/0.9% performance/DRAM energy
overheads and at $P_{mark}$ of 20% the overheads become only 11.3%/8.0%. is
more efficient than PARA mainly due to enabling access to non-locked regions
in a bank while performs neighbor row refreshes on the locked region.

We conclude that is a highly-efficient RowHammer protection that incurs small
performance and DRAM energy overheads even with high neighbor row refresh
probability, which is critical to protect future DRAM chips that may have
extremely high RowHammer vulnerability.

#####  5.6.4.6  Maximum Activation Threshold

We analyze the ’s sensitivity to the maximum row activation threshold
($ACT_{max}$). Fig. 5.16 shows the average speedup that and achieve for
different $ACT_{max}$ values across 4c-high workloads compared to the DDR4
baseline. When evaluating , we use as a DRAM refresh mechanism.

?figurename? 5.16: ’s sensitivity to $ACT_{max}$.

We observe that incurs negligible performance overhead on top of even for
extremely small $ACT_{max}$ values. This is because generates very few
neighbor row refreshes as 4c-high is a set of benign workloads that do not
repeatedly activate a single row many times.

Although the performance overhead of is negligible, the number of Counter
Table (CT) entries required is significantly large for small $ACT_{max}$
values. For $ACT_{max}=16K$, requires $38$ counters per bank, and the number
of counters required increase linearly as $ACT_{max}$ reduces, reaching $2449$
counters at the lowest $ACT_{max}=256$ that we evaluate.

#####  5.6.4.7 Number of Vulnerable Neighbor Rows

We analyze the performance overheads of and when refreshing a different number
of neighbor rows upon detecting a potentially aggressor row. Kim et al. [70]
show that, in some DRAM chips, an aggressor row can cause bit flips also in
rows that are at a greater distance than the two victim rows surrounding the
aggressor row. Thus, it may be desirable to configure a RowHammer protection
mechanism to refresh more neighbor rows than the two rows that are immediately
adjacent to the aggressor row.

Fig. 5.17 shows the average speedup that -Combined (separately with and )
achieves for different number of neighbor rows refreshed across 4c-high
workloads compared to the DDR4 baseline. The _Neighbor Row Distance_ values on
the x-axis represent the number of rows refreshes on each of the two sides of
an aggressor row (e.g., for neighbor row distance of $2$, and refresh four
victim rows in total).

?figurename? 5.17: Sensitivity to the number of neighbor rows affected by
RowHammer.

We make two key observations from the figure. First, incurs large performance
overheads as the neighbor row distance increases. This is because, with
$P_{mark}=1\%$, perform neighbor row refresh approximately for every 100th ACT
command, and the latency of this refresh operation increases with the increase
in the number of victim rows. Second, the performance overhead of is
negligible even when the neighbor row distance is five. This is because,
detects aggressor rows with a higher precision than using area-expensive
counters. As the 4c-high workloads do not repeatedly activate any single row,
counters rarely exceed the maximum activation threshold, and thus trigger
neighbor row refresh only a few times.

####  5.6.5 Hardware Overhead

An chip introduces an extra physical pin to transmit signals from a DRAM chip
to a Memory Controller (MC). For systems that use rank-based DRAM chip
organizations (e.g., DDR4 DIMM), the total number of extra pins depends on the
number of chips per rank. A typical ECC DIMM with 18 x4 DDR4 chips requires 18
pins per channel. The following two approaches can be used to reduce the pin
count for rank-based organizations. First, the RCD chip on Registered DIMMs
can act as a proxy between the DRAM chips and the MC by sending the MC a
single when any of the per-chip signals are asserted. A similar approach is
used with the per-chip alert_n signals in existing DDR4 modules. This approach
requires a single pin per channel. Second, can use the alert_n pin to transmit
an . alert_n is currently used to inform the MC that the DRAM chip detected a
CRC or parity check failure on the issued command, and thus the MC must issue
the command again. The alert_n signal can simply be asserted not only on CRC
or parity check failure, but also when the MC attempts accessing a row in a
locked region. This approach does not require additional physical pins.
Several DRAM standards (e.g., LPDDR, HBM, and GDDR) are typically organized as
a single chip per channel. For such memories, introduces only one pin per
channel.

We individually discuss the other changes required on the existing DRAM chip
and the MC circuitry.

#####  5.6.5.1 DRAM Chip Modifications

We use CACTI [384] to evaluate the hardware overhead of the changes that
introduces over a conventional DRAM bank (highlighted in Fig. 5.2) assuming
$22\text{\,}\mathrm{nm}$ technology. Lock Region Table (LRT) is a small table
that stores a single bit for each lock region to indicate whether or not the
lock region is under maintenance. In our evaluation, we assume that a bank is
divided into $16$ lock regions. Therefore, LRT consists of only $16$ bits,
which are indexed using a 4-bit lock region address. According to our
evaluation, an LRT incurs _only_ $32\text{\,}{\mathrm{\SIUnitSymbolMicro
m}}^{2}$ area overhead per bank. The area overhead of all LRTs in a DRAM chip
is _only_ 0.001% of a $45.5\text{\,}{\mathrm{mm}}^{2}$ DRAM chip. The access
time of an LRT is $0.053\text{\,}\mathrm{ns}$, which is _only_ 0.4% of typical
row activation latency (tRCD) of $13.5\text{\,}\mathrm{ns}$.

adds a per-region RA-latch to enable accessing one lock region while another
is under maintenance. An RA-latch stores a pre-decoded row address, which is
provided by the global row address decoder, and drives the local row decoders
where the row address is fully decoded. According to our evaluation, all RA-
latches incur a total area overhead of 1.63% of a
$45.5\text{\,}{\mathrm{mm}}^{2}$ DRAM chip. An RA-latch has only
$0.028\text{\,}\mathrm{ns}$ latency, which is negligible compared to tRCD.

Besides these changes that are the core of the substrate, a particular
maintenance mechanism may incur additional area overhead. We evaluate the DRAM
chip area overhead of the maintenance mechanisms presented in section 5.4. The
simple refresh mechanism, , requires only
$77.1\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{2}$ additional area in a DRAM
chip. maintains bloom filters that constitute a large portion of the required
$31\,663\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{2}$. The DRAM chip area
overhead of each refresh mechanism is less than 0.1% of a typical
$45.5\text{\,}{\mathrm{mm}}^{2}$ DRAM chip. requires
$863.6\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{2}$ area, which is a similarly
small portion of a typical DRAM chip. requires a large _Counter Table_ with
1224 counters per bank for the RowHammer threshold value $ACT_{max}=512$ that
we use in our performance evaluation. Across a DRAM chip, requires
$3.2\text{\,}{\mathrm{mm}}^{2}$ area, which is 7.0% of a typical DRAM chip
size. The control logic of is similar to the control logic of and it requires
only $77.1\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{2}$ additional area
excluding the area of the ECC engine, which is already implemented by DRAM
chips that support in-DRAM ECC.

#####  5.6.5.2 Memory Controller Modifications

We slightly modify the MC’s request scheduling mechanism to retry a rejected
ACT command as we explain in section 5.3.2. Upon receiving an , the MC marks
the corresponding bank as precharged. An existing MC already implements
control circuitry to pick an appropriate request from the request queue and
issue the necessary DRAM command based on the DRAM bank state (e.g., ACT to a
precharged bank or RD if the corresponding row is already open) by respecting
the DRAM timing parameters. The _ACT Retry Interval (ARI)_ is simply a new
timing parameter that specifies the minimum time interval for issuing an ACT
to a lock region after receiving an from the same region. Therefore, can be
implemented in existing MCs with only slight modifications by leveraging the
existing request scheduler (e.g., FRFCFS [172]).

No further changes are required in the MC to support different maintenance
mechanisms enabled by . Thus, enables the DRAM vendors to update existing
maintenance mechanisms and implement new ones without any further changes to
MCs that support .

###  5.7 Related Work

To our knowledge, this is the first paper to enable DRAM chips that
autonomously and efficiently perform various maintenance operations with
simple changes to existing DRAM interfaces. Our new () design enables new
maintenance operations completely within DRAM without requiring further
changes in the DRAM interface, Memory Controller (MC) or other system
components. No prior work proposes setting the MC free from managing DRAM
maintenance operations nor studies the system-level performance and energy
impact of autonomous maintenance mechanisms. We briefly discuss relevant prior
works.

Changing the DRAM Interface. Several prior works [385, 386, 387, 388] propose
using high-speed serial links and packed-based protocols in DRAM chips.
differs from prior works in two key aspects. First, none of these works
describe how to implement maintenance mechanisms completely within DRAM. We
propose six new -based maintenance mechanisms (section 5.4). Second, prior
works significantly overhaul the existing DRAM interface, which makes their
proposals more difficult to adopt compared to , which adds only a single
signal to the existing DRAM interface and requires slight modifications in the
MC.

Mitigating DRAM Refresh Overhead. Many previous works [41, 43, 49, 46, 50, 51,
53, 54, 55, 189, 162, 367, 302, 190, 57, 45, 58, 193, 59, 66, 68, 69, 191,
122, 192, 124, 56, 181, 368] propose techniques to reduce DRAM refresh
overhead. not only enables maintenance mechanisms for reducing DRAM refresh
overhead but also other efficient maintenance mechanisms for improving DRAM
reliability and security, described in section 5.4.

RowHammer Protection. Many prior works [85, 87, 78, 79, 88, 389, 84, 77, 73,
100, 390, 83, 81, 82, 73, 78, 94, 95, 96] propose techniques for RowHammer
protection. Although we propose and , can be used to implement existing or new
RowHammer protection mechanisms.

Memory Scrubbing. Although prior works report that the overhead of memory
scrubbing is small as low scrubbing rate (e.g., scrubbing period of 24 hours
[365, 366, 44], 45 minutes per 1 GB scrubbing rate [197], scrubbing only when
idle [196]) is generally sufficient, the cost of scrubbing can dramatically
increase for future DRAM chips due to increasing DRAM bit error rate and
increasing DRAM chip density. enables efficient scrubbing by eliminating off-
chip data transfers.

###  5.8 Summary

To set the memory controller free from managing DRAM maintenance operations,
() introduces minimal changes to the existing DRAM chips and memory
controllers. enables in-DRAM maintenance operations with no further changes to
the DRAM interface, memory controller, or other system components. Using , we
implement efficient maintenance mechanisms for DRAM refresh, RowHammer
protection, and memory scrubbing. We show that these mechanisms altogether
enable a higher performance, more energy efficient and at the same time more
reliable and secure DRAM system. To foster research and development in this
direction, we open source our Self-Managing DRAM framework [120] and early-
release our Self-Managing DRAM work on arXiv.org [121]. We believe and hope
that will enable practical adoption of innovative ideas in DRAM design.

##  ?chaptername? 6 CROW: A Low-Cost Substrate for Improving DRAM Performance,
Energy Efficiency, and Reliability

DRAM has been the dominant technology for architecting main memory for
decades. Recent trends in multi-core system design and large-dataset
applications have amplified the role of DRAM as a critical system bottleneck.
We propose Copy-Row DRAM (CROW), a flexible substrate that enables new
mechanisms for improving DRAM performance, energy efficiency, and reliability.
We use the CROW substrate to implement 1) a low-cost in-DRAM caching mechanism
that lowers DRAM activation latency to frequently-accessed rows by 38% and 2)
a mechanism that avoids the use of short-retention-time rows to mitigate the
performance and energy overhead of DRAM refresh operations. CROW’s flexibility
allows the implementation of both mechanisms at the same time. Our evaluations
show that the two mechanisms synergistically improve system performance by
20.0% and reduce DRAM energy by 22.3% for memory-intensive four-core
workloads, while incurring 0.48% extra area overhead in the DRAM chip and
$11.3\text{\,}\mathrm{KiB}$ storage overhead in the memory controller, and
consuming 1.6% of DRAM storage capacity, for one particular implementation. To
aid future research and development, we open source CROW [123].

###  6.1 Motivation and Goal

As the memory demands of applications have been growing, manufacturers have
been scaling the DRAM process technology to keep pace. Unfortunately, while
the density of the DRAM chips has been increasing as a result of scaling, DRAM
faces three critical challenges in meeting application demands [6, 5]: (1)
high access latencies and (2) high refresh overheads, both of which degrade
system performance and energy efficiency; and (3) increasing exposure to
vulnerabilities, which reduces the reliability of DRAM.

First, the high DRAM access latency is a challenge with respect to improving
system performance and energy efficiency. While DRAM capacity increased
significantly over the last two decades [7, 8, 9, 10, 2, 11], DRAM access
latency decreased only slightly [2, 6, 10]. The high DRAM access latency
significantly degrades the performance of many workloads [18, 19, 20, 21, 22].
The performance impact is particularly large for applications that 1) have
working sets exceeding the cache capacity of the system, 2) suffer from high
instruction and data cache miss rates, and 3) have low memory-level
parallelism. While manufacturers offer latency-optimized DRAM modules [37,
38], these modules have significantly lower capacity and higher cost compared
to commodity DRAM [2, 39, 3]. Thus, reducing the high DRAM access latency
_without trading off capacity and cost_ in commodity DRAM remains an important
challenge [23, 2, 5].

Second, the high DRAM refresh overhead is a challenge to improving system
performance and energy consumption. A DRAM cell stores data in a capacitor
that leaks charge over time. To maintain correctness, _every_ DRAM cell
requires periodic _refresh_ operations that restore the charge level in a
cell. As the DRAM cell size decreases with process technology scaling, newer
DRAM devices contain more DRAM cells than older DRAM devices [40]. As a
result, while DRAM capacity increases, the performance and energy overheads of
the refresh operations scale unfavorably [41, 42, 43]. In modern LPDDR4 [188]
devices, the memory controller refreshes _every_ DRAM cell every
$32\text{\,}\mathrm{ms}$. Previous studies show that 1) refresh operations
incur large performance overheads, as DRAM cells _cannot_ be accessed when the
cells are being refreshed [41, 43, 45, 46]; and 2) up to 50% of the total DRAM
energy is consumed by the refresh operations [43, 41].

Third, the increasing vulnerability of DRAM cells to various failure
mechanisms is an important challenge to maintaining DRAM reliability. As the
process technology shrinks, DRAM cells become smaller and get closer to each
other, and thus become more susceptible to failures [61, 5, 62, 63, 64, 43,
65]. A tangible example of such a failure mechanism in modern DRAM is
RowHammer [73, 72, 74, 76]. RowHammer causes _disturbance errors_ (i.e., bit
flips in vulnerable DRAM cells that are not being accessed) in DRAM rows
physically adjacent to a row that is repeatedly activated many times.

These three challenges are difficult to solve efficiently by directly
modifying the underlying cell array structure. This is because commodity DRAM
implements an extremely dense _DRAM cell array_ that is optimized for _low
area-per-bit_ [2, 11, 9]. Because of its density, even a small change in the
DRAM cell array structure may incur non-negligible area overhead [2, 391, 132,
5]. Our goal in this chapter is to lower the DRAM access latency, reduce the
refresh overhead, and improve DRAM reliability with _no changes_ to the DRAM
cell architecture, and with only _minimal_ changes to the DRAM chip.

###  6.2 Copy-Row DRAM

To efficiently solve the challenges of 1) high access latency, 2) high refresh
overhead, and 3) increasing vulnerability to failure mechanisms in DRAM,
without requiring any changes to the DRAM cell array, we introduce _Copy-Row
DRAM_ (CROW). CROW is a new, practical substrate that exploits the existing
DRAM architecture to efficiently duplicate a small number of rows in a
subarray at runtime. CROW is versatile, and enables new mechanisms for
improving DRAM performance, energy efficiency, and reliability, as we show in
section 6.3.

####  6.2.1 CROW: A High-Level Overview

CROW consists of two key components: 1) a small number of _copy rows_ in each
subarray, which can be used to duplicate or remap the data stored in one of
the remaining rows (called _regular rows_) of the subarray, 2) a table in the
memory controller, called the _CROW table_ , that tracks which rows are
duplicated or remapped.

CROW divides a DRAM subarray into two types of rows: regular rows and copy
rows. A copy row is similar to a regular row in the sense that it has the same
row width and DRAM cell structure. However, copy rows have their own small
local row decoder within the subarray, separate from the existing local row
decoder for regular rows. This enables the memory controller to activate copy
rows independently from regular rows (which continue to use the existing local
row decoder). By allowing copy rows and regular rows to be activated
independently, we enable two DRAM primitives that make use of _multiple-row
activation_ (MRA) in CROW.

First, CROW can perform _bulk data movement_ , where the DRAM copies an entire
row of data at once from a regular row to a copy row. To do so, the memory
controller first activates the regular row, and next activates the copy row,
immediately after the local row buffer latches the data of the regular row.
After the second activation, the sense amplifiers restore the data initially
stored only in the regular row to both the regular row and the copy row
(similar to the RowClone [128] mechanism).

Second, CROW can perform _reduced-latency DRAM access_. When a regular row and
a copy row contain _the same data_ , the memory controller activates both rows
_simultaneously_. This causes two cells on the same bitline to inject charge
into the bitline at a faster total rate than a single cell can. Thereby, the
sense amplifier to operate faster, which leads to lower activation latency via
an effect similar to increasing the amount of charge stored in a single cell
[179].

CROW makes use of a table in the memory controller, CROW table, that tracks
which regular rows are duplicated or mapped to copy rows. A mechanism that
takes advantage of the CROW substrate checks the CROW table and uses the
information in the table to either 1) simultaneously activate duplicate
regular and copy rows or 2) activate the copy row that a regular row is
remapped to. For example, the CROW-cache mechanism (section 6.3.1) updates a
CROW table entry with the address of the regular row that has been copied to
the corresponding copy row. Prior to issuing an ACT command to activate a
target regular row, the memory controller queries the CROW table to check
whether the regular row has a duplicate copy row. If so, the memory controller
issues a custom reduced-latency DRAM command to simultaneously activate both
the regular row and the copy row, instead of a single activate to the regular
row. Doing so enables faster access to the duplicated data stored in both
rows. Next, we explain CROW in detail.

####  6.2.2 Copy Rows

CROW logically categorizes the rows in a subarray into two sets, as shown in
Figure 6.1: (1)  _regular rows_ , which operate the same as in conventional
DRAM; and (2)  _copy rows_ , which can be activated independently of the
regular rows. In our design, we add a small number of copy rows to the
subarray, and add a second local row decoder (called the _CROW decoder_) for
the copy rows.111Alternatively, CROW can use a small set of the existing rows
in a conventional subarray as copy rows. Because the CROW decoder drives a
much smaller number of rows than the existing local row decoder, it has a much
smaller area cost (section 6.5.2). The memory controller provides the regular
row and copy row addresses to the respective decoders when issuing an ACT
command.

?figurename? 6.1: Regular and copy rows in a subarray in CROW.

####  6.2.3 CROW Table

As shown in Figure 6.2, the CROW table in the memory controller stores
information on 1) whether a copy row is allocated, and 2) which regular row is
duplicated or remapped to a copy row. For example, in our weak row remapping
scheme (section 6.3.2), a CROW table entry holds the address of the regular
row that the copy row replaces. The CROW table stores an entry for each copy
row in a DRAM channel, and is _n_ -way set associative, where _n_ is the
number of copy rows in each subarray.

?figurename? 6.2: Organization of the CROW table.

The memory controller indexes the CROW table using a combination of bank and
subarray addresses, which are part of the physical address of a memory
request. However, in current systems, subarray addresses are not exposed to
the memory controller. Therefore, to enable CROW, it is required to expose
such information to the memory controller.

The memory controller checks the table for an entry associated with the
regular row that the memory request is to access. Our current design stores
three fields in each CROW table entry: 1) the Allocated field indicates
whether the entry is valid or not; 2) the RegularRowID field stores a pointer
to the regular row that the corresponding copy row is associated with 3) the
Special field stores additional information specific to the mechanism that is
implemented using CROW.

###  6.3 Applications of CROW

CROW is a versatile substrate that enables multiple mechanisms for improving
DRAM performance, energy efficiency, and reliability. We propose three such
new mechanisms based on CROW: 1) CROW-cache, a mechanism that reduces DRAM
access latency by exploiting CROW’s ability to simultaneously activate a
regular row and a copy row that store the same data; 2) CROW-ref, a mechanism
that reduces DRAM refresh overhead by exploiting CROW’s ability to remap weak
rows with low retention times to strong copy rows. This mechanism relies on
retention time profiling [122, 49, 65] to determine weak and strong rows in
DRAM; 3) a mechanism that protects against the RowHammer [73] vulnerability by
identifying rows that are vulnerable to RowHammer-induced errors and remapping
the vulnerable rows to copy rows.

####  6.3.1 In-DRAM Caching (CROW-cache)

Our in-DRAM caching mechanism, CROW-cache, exploits both of our new multiple-
row activation primitives in CROW (section 6.2.1). The key idea of CROW-cache
is to use a copy row as a duplicate of a recently-accessed regular row within
the same subarray, and to activate _both_ the regular row and the copy row. By
simultaneously activating both rows, CROW-cache reduces activation latency and
starts servicing read and write requests sooner.

#####  6.3.1.1 Copying a Regular Row to a Copy Row

Given $N$ copy rows, we would like to duplicate the $N$ regular rows that will
be most frequently activated in the near future to maximize the benefits of
CROW-cache. However, determining the $N$ most frequently-activated rows is a
difficult problem, as applications often access main memory with irregular
access patterns. Therefore, we follow a similar approach used by processor
caches, where the most-recently-used rows are cached instead of the most
frequently-accessed rows. Depending on how many copy rows are available, CROW-
cache maintains copies of the $N$ _most-recently-activated_ regular rows in
each subarray. When the memory controller needs to activate a regular row that
is not already duplicated, the memory controller copies the regular row to an
available copy row, using multiple-row activation. While this is a simple
scheme, we achieve a significant hit rate on the CROW table with a small
number of copy rows (see section 6.7.1.1).

To efficiently copy a regular row to a copy row within a subarray, we
introduce a new DRAM command, which we call _Activate-and-copy_ (ACT-c). The
ACT-c command performs a three-step row copy operation in DRAM, using a
technique similar to RowClone [128]. First, the memory controller issues ACT-c
to DRAM and sends the addresses of 1) the _source_ regular row and 2) the
_destination_ copy row, over multiple command bus cycles. Second, upon
receiving an ACT-c command, DRAM enables the wordline of only the regular row.
The process of reading and latching the row’s data into the sense amplifiers
completes as usual (see section 2.2). Third, as soon as the data is latched in
the sense amplifiers, DRAM enables the wordline of the copy row. This causes
the sense amplifiers to restore charge to both the regular row _and_ the copy
row. On completion of the restoration phase, both the regular row and the copy
row contain the same data.

The ACT-c operation slightly increases the restoration time compared to
regular ACT. This is because, after the wordline of the copy row is enabled,
the capacitance that the sense amplifier drives the bitline against increases,
since two cell capacitors are connected to the same bitline. In our circuit-
level SPICE model, for the ACT-c command, we find that the activate-to-
precharge latency (tRAS) increases by 18%. However, this activation latency
overhead has a very limited impact on overall system performance because CROW
table typically achieves a high hit rate (section 6.7.1.1), which means that
the latency of duplicating the row is amortized by the reduced activation
latency of future accesses to the row.

#####  6.3.1.2 Reducing Activation Latency

CROW-cache introduces a second new DRAM command, _Activate-two_ (ACT-t), to
simultaneously activate a regular row and its duplicate copy row. If the CROW
table (see section 6.2.3) contains an entry indicating that a copy row is a
duplicate of the regular row to be activated, the memory controller issues
ACT-t to perform MRA on both rows, achieving low-latency activation. In
section 6.4, we perform detailed circuit-level SPICE simulations to analyze
the activation latency reduction with ACT-t.

Note that the modifications needed for ACT-c and ACT-t in the row decoder
logic are nearly identical, as they both perform multiple-row activation. The
difference is that rather than activating the copy row _after_ the regular row
as with ACT-c, ACT-t activates both of the rows concurrently.

#####  6.3.1.3 Reducing Restoration Latency

In addition to decreasing the amount of time required for charge sharing
during activation (see section 2.2), the increased capacitance on the bitline
due to the activation of multiple cells enables the reduction of tRAS by using
_partial restoration_ [136, 183]. The main idea of the partial restoration
technique is to terminate row activation earlier than normal, i.e., issue PRE
by relaxing (i.e., reducing) tRAS, on a multiple-row activation such that the
cells are not fully restored. While this degrades the retention time of each
cell individually, the partially restored cells contain just enough charge to
maintain data integrity until the next refresh operation [192] (we verify this
using SPICE simulations in section 6.4). We combine partial cell restoration
(i.e., reduction in tRAS) with the reduction in tRCD in order to provide an
even greater speedup than possible by simply decreasing tRCD alone (see
section 6.7).

Taking the concept one step further, we can also apply partial restoration to
write accesses, decreasing the amount of time required for the WR command.
Since the write restoration latency (tWR) is analogous to cell restoration
during activation (section 2.2), we can terminate the write process early
enough such that the written cells are only partially charged.

#####  6.3.1.4 Evicting an Entry from the CROW Table

As described in section 6.3.1.3, CROW-cache relaxes tRAS for the ACT-t command
to further improve the DRAM access latency. Relaxing tRAS may terminate the
restoration operation early, which puts the precharged regular and copy rows
into a _partially-restored_ state.

Note that there are two cases where relaxing tRAS does not necessarily lead to
partial restoration. First, the memory controller can exploit an open regular
row + copy row pair to serve multiple memory requests, if available in the
request queue, from different portions of the row. The time needed to serve
these requests can delay precharging the associated regular and copy rows
until they are fully restored. Second, if there are no other memory requests
to different rows in the same bank, the currently-opened regular row and copy
row can be left open, providing enough time for full charge restoration.

CROW-cache maintains the restoration state of the paired regular and copy rows
by utilizing a single bit of the Special field of the CROW table, which in the
context of CROW-cache we refer to as the isFullyRestored field. CROW-cache
sets isFullyRestored to false only if 1) ACT-t was used to activate the
currently open row, and 2) the memory controller issues a PRE command to close
the rows _before_ the default tRAS is satisfied. In contrast, CROW-cache sets
isFullyRestored to true if the memory controller issues the next PRE after the
default tRAS (i.e., a time interval sufficient for fully restoring the open
regular row and copy row). Note that existing memory controllers already
maintain timing information for conventional command scheduling, e.g., when
the last ACT to each bank was issued. By taking advantage of the existing
timing information, CROW-cache does not incur significant area overhead for
managing the isFullyRestored field.

Partially restoring the duplicate rows helps to significantly reduce
restoration latency (see section 6.4). However, partial restoration introduces
a new challenge to the design of CROW-cache, because a partially-restored row
can only be correctly accessed using ACT-t, which activates a regular row
along with its duplicate copy row. Duplicating a new regular row ($RR_{new}$)
to a copy row that is already a duplicate of another regular row ($RR_{old}$),
i.e., evicting $RR_{old}$ from the CROW table, causes $RR_{old}$ to be
activated as a single row during a future access. If $RR_{old}$ is partially
restored prior to eviction from the CROW table, performing a single-row
activation on $RR_{old}$ in the future may lead to data corruption. Thus, the
memory controller needs to guarantee that a partially-restored regular row is
never evicted from the CROW table.

To prevent data corruption due to the eviction of a partially-restored row
from the CROW table, if the _isFullyRestored_ field is _false_ , the memory
controller first issues an ACT-t and fully restores $RR_{old}$ by conforming
to the _default_ tRAS. Since this operation sets _isFullyRestored_ to _true_ ,
$RR_{old}$ can now safely be evicted from the CROW table to be replaced with
$RR_{new}$. The disadvantage of this approach is the overhead of issuing an
additional ACT-t followed by a PRE to perform full restoration. However, in
section 6.7.1.1, we show that this overhead has a negligible performance
impact since CROW table has a high hit rate.

#####  6.3.1.5 Implementing the New DRAM Commands

To implement CROW-cache, we introduce the ACT-c and ACT-t DRAM commands, both
of which activate a regular row _and_ a copy row. The wordline of a regular
row is enabled in the same way as in conventional DRAM. To enable the wordline
of a copy row, we extend the existing local row decoder logic such that it can
drive the wordline of a copy row independently of the regular row. As we show
in section 6.5.2, for the default configuration of the CROW substrate with
eight copy rows, our modifications increase the decoder area by 4.76%, leading
to 0.48% area overhead in the entire DRAM chip. The additional eight copy rows
per subarray require only 1.6% of the DRAM storage capacity.

Unlike ACT, which specifies a single row address, ACT-c and ACT-t need to
specify the addresses of both a regular and a copy row. ACT-c and ACT-t need
only a small number of copy row address bits in addition to the regular row
address bits since the corresponding copy row is in the same subarray as the
activated regular row. For CROW-8, where each subarray has eight copy rows, we
need only three additional bits to encode the copy row address. We can either
add three wires to the address bus (likely undesirable) or send the address
over multiple cycles as done in current LPDDR4 chips for the ACT, RD, and WR
commands [177].222In our evaluations, we assume an additional cycle on the
command/address bus to send the copy row address to DRAM. This additional
cycle does not always impact the activation latency, as the memory controller
can issue the ACT-c and ACT-t commands one cycle earlier if the
command/address bus is idle. Also, DRAM does not immediately need the address
of the copy row for ACT-c.

####  6.3.2 Reducing Refresh Overhead (CROW-ref)

To reduce the DRAM refresh overhead, we take advantage of the CROW substrate
to develop CROW-ref, a software-transparent weak row remapping scheme. CROW-
ref extends the refresh interval of the _entire_ DRAM chip beyond the worst-
case value defined in DRAM standards ($64\text{\,}\mathrm{ms}$ for DDR3 and
DDR4, $32\text{\,}\mathrm{ms}$ for LPDDR4) by avoiding the use of the very
small set of weak rows in a given DRAM chip, i.e., rows that would fail when
the refresh interval is extended (due to the existence of at least one weak
cell in the row). CROW-ref consists of three key components. First, CROW-ref
uses retention time profiling at system boot or during runtime to identify the
weak rows. Second, CROW-ref utilizes the strong copy rows provided by CROW in
each subarray to store the data that would have originally been stored in weak
regular rows. Third, CROW-ref uses the CROW table to maintain the remapping
information, i.e., which strong copy row replaces which weak regular row.

#####  6.3.2.1 Identifying Weak Rows

To identify the weak rows in each subarray, we rely on retention time
profiling methodologies proposed by prior work [122, 49, 65, 66, 68, 67, 73,
212, 63]. A retention time profiler tests the DRAM device with various data
patterns and a wide range of operating temperatures to cover all DRAM cells
that fail at a chosen target retention time. Prior works find that very few
DRAM cells fail when the refresh interval is extended by 2x-4x. For example,
Liu et al. [65] show that _only_ $\sim$1000 cells in a
$32\text{\,}\mathrm{GiB}$ DRAM module fail when the refresh interval is
extended to $256\text{\,}\mathrm{ms}$. Assuming the experimentally-
demonstrated uniform random distribution of these weak cells in a DRAM chip
[122, 49, 65, 43, 50], we calculate that the _bit error rate (BER)_ is
$\leavevmode\nobreak\ 4\cdot 10^{-9}$ when operating at a
$256\text{\,}\mathrm{ms}$ refresh interval.

Based on this observation, we can determine how effective CROW-ref is likely
to be for a given number of copy rows. First, using the BER, we can calculate
$P_{\text{\em weak\\_row}}$, the probability that a row contains at least one
weak cell, as follows:

$P_{\text{\em weak\\_row}}=1-(1-BER)^{N_{\text{\em cells\\_per\\_row}}}$ (6.1)

where $N_{\text{\em cells\\_per\\_row}}$ is the number of DRAM cells in a row.
Second, we can use $P_{\text{\em row}}$ to determine $P_{\text{\em
subarray}-n}$, which is the probability that a subarray with $N_{\text{\em
rows}}$ rows contains more than $n$ weak rows:

$P_{\text{\em subarray}-n}=1-\sum\limits_{k=0}^{n}\binom{N_{\text{\em
rows}}}{k}P_{\text{\em row}}^{k}(1-P_{\text{\em row}})^{N_{\text{\em
rows}}-k}$ (6.2)

Using these equations, for a DRAM chip with 8 banks, 128 subarrays per bank,
512 rows per subarray, and $8\text{\,}\mathrm{KiB}$ per row, the probability
of any subarray having more than 1/2/4/8 weak rows is $0.99/3.1\times
10^{-1}/3.3\times 10^{-4}/3.3\times 10^{-11}$. We conclude that the
probability of having a subarray with a large number of weak rows is extremely
low, and thus CROW-ref is highly effective even when the CROW substrate
provides only 8 copy rows per subarray. In the unlikely case where the DRAM
has a subarray with more rows with weak cells than the number of available
copy rows, CROW-ref falls back to the default refresh interval, which does not
provide performance and energy efficiency benefits but ensures correct DRAM
operation.333Alternatively, CROW-ref can be combined with a heterogeneous
refresh-rate scheme, e.g., RAIDR [43] or AVATAR [49].

#####  6.3.2.2 Operation of the Remapping Scheme

In each subarray, CROW-ref remaps weak regular rows to _strong_ copy rows in
the same subarray.444We do _not_ use a weak copy row to replace a weak regular
row, as a weak copy row would also not maintain data correctly for an extended
refresh interval. The retention time profiler also identifies the weak copy
rows. CROW-ref tracks the remapping using the CROW table. When a weak regular
row is remapped to a strong copy row, CROW-ref stores the row address of the
regular row into the RegularRowID field of the CROW table entry that
corresponds to the copy row. When the memory controller needs to activate a
row, it checks the CROW table to see if any of the RegularRowID fields
contains the address of the row to be activated. If one of the RegularRowID
fields matches, the memory controller issues an activate command to the copy
row that replaces the regular row, instead of to the regular row. On a CROW
table miss, which indicates that the row to be activated contains only strong
cells, the memory controller activates only the original regular row. This row
remapping operates transparently from the software, as the regular-to-copy-row
remapping is _not_ exposed to the software.

#####  6.3.2.3 Support for Dynamic Remapping

CROW-ref can dynamically change the remapping of weak rows if new weak rows
are detected at runtime. This functionality is important as DRAM cells are
known to be susceptible to a failure mechanism known as _variable retention
time_ (VRT) [49, 122, 65, 67, 68, 212, 63, 217]. As a VRT cell can
nondeterministically transition between high and low retention states, new VRT
cells need to be continuously identified by periodically profiling DRAM, as
proposed by prior work [49, 122, 66]. To remap a newly-identified weak regular
row at runtime, the memory controller simply allocates an unused strong copy
row from the subarray, and issues an ACT-c to copy the regular row’s data into
the copy row. Once this is done, the memory controller accesses the strong
copy row instead of the weak regular row, as we explain in section 6.3.2.2.

####  6.3.3 Mitigating RowHammer

We propose a third mechanism enabled by the CROW substrate that protects
against the RowHammer vulnerability [73, 72, 74]. Due to the small DRAM cell
size and short distance between DRAM cells, electromagnetic coupling effects
that result from rapidly activating and precharging (i.e., hammering) a DRAM
row cause bit flips on the physically-adjacent (i.e., victim) rows [73, 72,
74]. This effect is known as _RowHammer_ , and has been demonstrated on real
DRAM chips [73]. Aside from the decreased reliability of DRAM due to
RowHammer, prior works (e.g., [233, 83, 235, 236, 77, 237, 238, 239, 240, 241,
242, 134, 243, 244, 245, 246, 100, 247, 248, 249, 250, 251, 252, 253, 254])
exploit RowHammer to perform attacks that gain privileged access to the
system. Therefore, it is important to mitigate the effects of RowHammer in
commodity DRAM.

We propose a CROW-based mechanism that mitigates RowHammer by remapping the
victim rows adjacent to the hammered row to copy rows. By doing so, the
mechanism prevents the attacker from flipping bits on the data originally
allocated in the victim rows. Several prior works [392, 85, 393, 195] propose
techniques for detecting access patterns that rapidly activate and precharge a
specific row. Typically, these works implement a counter-based structure that
stores the number of ACT commands issued to each row. Our RowHammer mitigation
mechanism can use a similar technique to detect a RowHammer attack. When the
memory controller detects a DRAM row that is being hammered, it issues two
ACT-c commands to copy the victim rows that are adjacent to the hammered row
to two of the available copy rows in the same subarray. Similar to the CROW-
ref mechanism (section 6.3.2), our RowHammer mitigation mechanism tracks a
remapped victim row using the RegularRowID field in each CROW table entry, and
looks up the CROW table every time a row is activated to determine if a victim
row has been remapped to a copy row.

CROW enables a simple yet effective mechanism for mitigating the RowHammer
vulnerability that can reuse much of the logic of CROW-ref. We leave the
evaluation of our RowHammer mitigation mechanism to future work.

###  6.4 Circuit Simulations

We perform detailed circuit-level SPICE simulations to find the latency of 1)
simultaneously activating two DRAM rows that store the same data (ACT-t) and
2) copying an entire regular row to a copy row inside a subarray (ACT-c).
Table 6.1 shows a summary of change in the tRCD, tRAS, and tWR timing
parameters that we use for the two new commands, based on the SPICE
simulations. We discuss in detail how the timings are derived for ACT-t in
section 6.4.1, and for ACT-c in section 6.4.2.

?tablename? 6.1: Timing parameters for new DRAM commands.

DRAM Command | tRCD | tRAS | tWR
---|---|---|---
ACT-t | _activating fully-restored rows_ | -38% | -7%ax (-33%bx) | +14%ax (-13%bx)
_activating partially-restored rows_ | -21% | -7%ax (-25%bx) | +14%ax (-13%bx)
ACT-c | 0% | +18%ax (-7%bx) | +14%ax (-13%bx)

  *   * a

When fully restoring the charge.

  * b

When terminating charge restoration early.

In our simulations, we model the entire cell array of a modern DRAM chip using
$22\text{\,}\mathrm{nm}$ DRAM technology parameters, which we obtain by
scaling the reference $55\text{\,}\mathrm{nm}$ technology parameters [394]
based on the ITRS roadmap [395, 40]. We use $22\text{\,}\mathrm{nm}$ PTM low-
power transistor models [396, 397] to implement the access transistors and the
sense amplifiers. In our SPICE simulations, we run $10^{4}$ iterations of
Monte-Carlo simulations with a 5% margin on every parameter of each circuit
component, to account for manufacturing process variation. Across all
iterations, we observe that the ACT-t and ACT-c commands operate correctly. We
report the latency of the ACT-t and ACT-c commands based on the Monte-Carlo
simulation iteration that has the highest access latency for each of these
commands. We release our SPICE model as an open-source tool [123].

####  6.4.1 Simultaneous Row Activation Latency

Simultaneously activating multiple rows that store the same data accelerates
the _charge-sharing_ process, as the increased amount of charge driven on each
bitline perturbs the bitline faster compared to single-row activation. Figure
6.3(a) plots the reduction in activation latency (tRCD) for a varying number
of simultaneously-activated rows. As seen in the figure, we observe a tRCD
reduction of 38% when simultaneously activating _two_ rows. tRCD reduces
further when we increase the number of simultaneously-activated rows, but the
latency reduction per additional activated row becomes smaller. We empirically
find that simultaneously activating only two rows rather than more rows
achieves a large portion of the maximum possible tRCD reduction potential
(i.e., as we approach an infinite number of rows being activated
simultaneously) with low area and power overhead (see section 6.5.2).

(a) tRCD ($18\text{\,}\mathrm{ns}$)

(b) tRAS ($42\text{\,}\mathrm{ns}$), tWR ($18\text{\,}\mathrm{ns}$), and
restoration ($24\text{\,}\mathrm{ns}$)

?figurename? 6.3: Change in various DRAM latencies with different number of
simultaneously-activated rows, normalized to baseline DRAM timing parameters
(absolute baseline values in parentheses).

Change in Restoration Latency. Although two-row activation reduces tRCD,
_fully_ restoring the capacitors of two cells takes more time compared to
fully restoring a single cell. Therefore, two-row activation can potentially
increase tRAS and tWR, which could offset the benefits of the tRCD reduction.
Figure 6.3(b) shows the change in tRAS, restoration time, and tWR for a
varying number of simultaneously-activated rows. Although restoration time
always increases with the number of rows, we see a slight decrease in tRAS
when the number of rows is small. This is because the reduction in tRCD, which
is part of tRAS (as we explain in section 2.2), is larger than the increase in
restoration time. However, for five or more simultaneously-activated rows, the
overhead in restoration time exceeds the benefits of tRCD reduction, and, as a
result, tRAS increases. tWR always increases with the number of
simultaneously-activated rows, because writing to a DRAM cell is similar to
restoring a cell (section 2.2).

Terminating Restoration Early. We use CROW to enable in-DRAM caching by
duplicating recently-accessed rows and using two-row activation to reduce tRCD
significantly for future activation of such rows. However, to make the in-DRAM
caching mechanism more effective, we aim to reduce tRAS further, as it is a
critical timing parameter that affects how quickly we can switch between rows
in the event of a row buffer conflict. We make _three_ observations based on
two-row activation, which leads us to a trade-off between reducing tRCD and
reducing tRAS. First, when two DRAM rows are used to store the same data, the
data is correctly retained for a longer period of time compared to when the
data is stored in a single row. This is due to the increased aggregate
capacitance that storing each bit of data using two cells provides. Second,
since data is correctly retained for a longer period of time when stored in
two rows, we can terminate the restoration operation early to reduce tRAS and
still achieve the target retention time, e.g., $64\text{\,}\mathrm{ms}$.
Third, as the amount of charge stored in a cell capacitor decreases, the
activation latency increases [17, 11]. Therefore, terminating restoration
early reduces tRAS at the expense of a slight increase in tRCD (due to less
charge).

We explore the trade-off space between reducing tRCD and reducing tRAS for a
varying number of simultaneously-activated rows using our SPICE model. In
Figure 6.4, we show the different tRCD and tRAS latencies that can be used
with multiple-row activation (MRA) while still ensuring data correctness. For
two-row activation, we empirically find that a 21% reduction in tRCD and a 33%
reduction in tRAS provides the best performance on average for the workloads
that we evaluate (section 6.7.1.1).

?figurename? 6.4: Normalized tRCD latency as a function of normalized tRAS
latency for different number of simultaneously activated DRAM rows.

We also eliminate the tWR overhead of MRA by terminating the restoration step
early when writing to simultaneously-activated DRAM rows. Since reducing tWR
increases the tRCD latency for the next activation of the row, there exists a
trade-off between reducing tRCD and reducing tWR, similar to the trade-off
between reducing tRCD and reducing tRAS. In our evaluations, we find that to
achieve a 21% reduction in tRCD, we can reduce tWR by 13% (see Table 6.1) by
terminating the restoration step early during a write to simultaneously-
activated DRAM rows.

####  6.4.2 MRA-Based Row Copy Latency

As we explain in section 6.3.1.1, CROW-cache uses the new ACT-c command to
efficiently copy a regular row to a copy row in the same subarray by
activating the copy row slightly after activating the regular row. This gives
the sense amplifiers sufficient time to correctly read the data stored in the
regular row. Using our SPICE model, we find that ACT-c does not affect tRCD
because the copy row is activated after satisfying tRCD. This is because, to
start restoring the data of the regular row to both the regular row and the
copy row, the local row buffer first needs to correctly latch the data of the
regular row. In contrast to tRCD, the ACT-c command increases tRAS by 18%
(reduces tRAS by 7% when terminating restoration early), as restoring two DRAM
rows requires more time than restoring a single row. We model this additional
time in all of our evaluations.

###  6.5 Hardware Overhead

Implementing the CROW substrate requires only a small number of changes in the
memory controller and the DRAM die.

####  6.5.1 Memory Controller Overhead

CROW introduces the CROW table (section 6.2.3), which incurs modest storage
overhead in the memory controller. The storage requirement for each entry
($Storage_{\text{\em entry}}$) in CROW table can be calculated in terms of
bits using the following equation:

$Storage_{\text{\em entry}}=\lceil log_{2}(RR)\rceil+Bits_{\text{\em
Special}}+Bits_{\text{\em Allocated}}$ (6.3)

where $RR$ is the number of regular rows in each subarray (we take the log of
$RR$ to represent the number of bits needed to store the RegularRowID field),
$Bits_{\text{\em Special}}$ is the size of the Special field in bits, and
$Bits_{\text{\em Allocated}}$ is set to 1 to indicate the single bit used to
track whether the entry is currently valid. Note that the RegularRowID field
does not have to store the entire row address. Instead, it is sufficient to
store a pointer to the position of the row in the subarray. For an example
subarray with 512 regular rows, an index range of 0–511 is sufficient to
address all regular rows in same subarray. We assume one bit for the Special
field, which we need to distinguish between the CROW-cache and CROW-ref
mechanisms (section 6.3).

We calculate $Storage_{\text{\em{CROW} table}}$, the storage requirement for
the entire CROW table, as:

$Storage_{\text{\em{CROW} table}}=Storage_{\text{\em entry}}*CR*SA$ (6.4)

where _CR_ is the number of copy rows in each subarray, and _SA_ is the total
number of subarrays in the DRAM. We calculate the storage requirement of the
CROW table for a single-channel memory system with 512 regular rows per
subarray, 1024 subarrays (8 banks; 128 subarrays per bank), and 8 copy rows
per subarray to be $11.3\text{\,}\mathrm{KiB}$. Thus, the processor die
overhead is small.

The CROW table storage overhead is proportional to the memory size. Although
the overhead is small, we can further optimize the CROW table implementation
to reduce its storage overhead for large memories. One such optimization
shares one set of CROW table entries across multiple subarrays. While this
limits the number of copy rows that can be utilized simultaneously, CROW can
still capture the majority of the benefits that would be provided if the CROW
substrate had a separate CROW table for each subarray. From our evaluations,
when sharing each CROW table entry between 4 subarrays (i.e., approximately a
factor of 4 reduction in CROW table storage requirements), we observe that the
average speedup that CROW-cache provides for single-core workloads reduces
from 7.1% to only 6.1%.

We evaluate the access time of a CROW table with the configuration given in
Table 6.2 using CACTI [384], and find that the access time is only
$0.14\text{\,}\mathrm{ns}$. We do _not_ expect the table access time to have
any impact on the overall cycle time of the memory controller.

####  6.5.2 DRAM Die Area Overhead

To activate multiple rows in the same subarray, we modify the row decoder to
drive multiple wordlines at the same time. These modifications incur a small
area overhead and cause MRA to consume more power compared to a single-row
activation.

In Figure 6.5 (left), we show the activation power overhead for simultaneously
activating up to nine DRAM rows in the same subarray. The ACT-c and ACT-t
commands simultaneously activate two rows, and consume only 5.8% additional
power compared to a conventional ACT command that activates only a single row.
The slight increase in power consumption is mainly due to the small copy row
decoder that CROW introduces to drive the copy rows.

?figurename? 6.5: Power consumption and area overhead of MRA.

Figure 6.5 (right) plots the area overhead of a copy row decoder that enables
one of the copy rows in the subarray independently from the existing regular
row decoder. The figure shows the copy row decoder area overhead as we vary
the number of copy rows in a subarray. For CROW-8, which has eight copy rows
per subarray, the decoder area increases by 4.8%, which corresponds to only a
0.48% area overhead for the entire DRAM chip. The area overhead of CROW-8 is
small because the decoder required for eight copy rows is as small as
$9.6\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{2}$, while our evaluations show
that the local row decoder for 512 regular rows occupies
$200.9\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{2}$.

###  6.6 Methodology

We use Ramulator [138, 300], a cycle-accurate DRAM simulator, to evaluate the
performance of the two mechanisms that we propose based on the CROW substrate.
We run Ramulator in CPU-trace-driven mode and collect application traces using
a custom Pintool [376]. The traces include virtual addresses that are accessed
by the application during the trace collection run. In Ramulator, we perform
virtual-to-physical address translation by randomly allocating a
$4\text{\,}\mathrm{KiB}$ physical frame for each access to a new virtual page,
which emulates the behavior of a steady-state system [398].

Table 6.2 provides the system configuration we evaluate. Unless stated
otherwise, we implement CROW with eight copy rows per subarray. We analyze the
performance of CROW-cache and CROW-ref on single- and multi-core system
configurations using typical LPDDR4 timing parameters, as shown in the table.
Although our evaluation is based on LPDDR4 DRAM, which is predominantly used
in various low-power systems [23], the mechanisms that we propose can also
reduce access latency in other DRAM-based memories, including 3D-stacked DRAM
[125].

?tablename? 6.2: Simulated system configuration. Processor |  1-4 cores, $4\text{\,}\mathrm{GHz}$ clock frequency, 4-wide issue, 8 MSHRs per core, 128-entry instruction window
---|---
Last-Level Cache |  $64\text{\,}\mathrm{B}$ cache-line, 8-way associative, $8\text{\,}\mathrm{MiB}$ capacity
Memory Controller |  64-entry read/write request queue, FR-FCFS-Cap555We use a variation of the FR-FCFS [143, 399] policy, called FR-FCFS-Cap [172], which improves fairness by enforcing an upper limit for the number of read/write requests that a row can service once activated. This policy performs better, on average, than the conventional FR-FCFS policy [143, 399], as also shown in [172, 171, 312, 174]. scheduling policy [172], timeout-based row buffer policy666The timeout-based row buffer management policy closes an open row after $75\leavevmode\nobreak\ ns$ if there are no requests in the memory controller queues to that row.
DRAM |  LPDDR4 [177], $1600\text{\,}\mathrm{MHz}$ bus frequency, 4 channels, 1 rank, 8 banks/rank, 64K rows/bank, 512 rows/subarray, $8\text{\,}\mathrm{KiB}$ row buffer size, tRCD/tRAS/tWR 29 (18)/67 (42)/29 (18) cycles (ns)

Workloads. We evaluate 44 single-core applications from four benchmark suites:
SPEC CPU2006 [377], TPC [378], STREAM [379], and MediaBench [380]. In
addition, we evaluate two synthetic applications [400] (excluded from our
average performance calculations): 1)  _random_ , which accesses memory
locations at random and has very limited row-level locality; and 2)
_streaming_ , which has high row-level locality because it accesses contiguous
locations in DRAM such that the time interval between two consecutive memory
requests is long enough for the memory controller to precharge the recently-
open row.

We classify the applications into three groups based on the misses-per-kilo-
instruction (MPKI) in the last-level cache. We obtain the MPKI of each
application by analyzing SimPoint [401] traces (200M instructions) of each
application’s representative phases using the single-core configuration. The
three groups are:

  * $\bullet$

L (low memory intensity): $MPKI<1$

  * $\bullet$

M (medium memory intensity): $1\leq MPKI<10$

  * $\bullet$

H (high memory intensity): $MPKI\geq 10$

We create eight multi-programmed workload groups for the four-core
configuration, where each group consists of 20 multi-programmed workloads.
Each group has a mix of workloads of different memory intensity classes. For
example, _LLHH_ indicates a group of 20 four-core multi-programmed workloads,
where each workload consists of two randomly-selected single-core applications
with low memory intensity (L) and two randomly-selected single-core
applications with high memory intensity (H). In total, we evaluate 160 multi-
programmed workloads. We simulate the multi-core workloads until each core
executes at least 200 million instructions. For all configurations, we
initially warm up the caches by fast-forwarding 100 million instructions.

Metrics. We measure the speedup for single-core applications using the
instructions per cycle (IPC) metric. For multi-core evaluations, we use the
weighted speedup metric [382], which prior work shows is a good measure of job
throughput [381].

We use CACTI [384] to evaluate the DRAM area and power overhead of our two
mechanisms (i.e., CROW-cache and CROW-ref) and prior works (i.e., TL-DRAM [2]
and SALP [3]) that we compare against. We perform a detailed evaluation of the
latency impact of MRA using our circuit-level SPICE model [123]. We use
DRAMPower [168] to estimate DRAM energy consumption of our workloads.

###  6.7 Evaluation

We evaluate the performance, energy efficiency, and area overhead of the CROW-
cache and CROW-ref mechanisms.

####  6.7.1 CROW-cache

We evaluate CROW-cache with different numbers of copy rows per subarray in the
CROW substrate. We denote each configuration of CROW in the form of
CROW-$C_{r}$, where $C_{r}$ specifies the number of copy rows (e.g., we use
CROW-8 to refer to CROW with eight copy rows). To show the potential of CROW-
cache, we also evaluate a hypothetical configuration with a 100% CROW table
hit rate, referred to as _Ideal CROW-cache_.

#####  6.7.1.1 Single-core Performance.

Figure LABEL:crow_fig:crow_sc_perf (top) shows the speedup of CROW-cache over
the baseline and the CROW table hit rate (bottom) for single-core
applications. We also list the last-level cache MPKI of each application to
indicate memory intensity. We make four observations from the figure.

First, CROW-cache provides significant speedups. On average, CROW-1, CROW-8,
and CROW-256 improve performance by 5.5%, 7.1%, and 7.8%, respectively, over
the baseline.777We notice that, for some applications (e.g., _jp2-encode_),
using fewer copy rows slightly outperforms a configuration with more copy
rows. We find the reason to be the changes in memory request service order due
to the fact that memory controller makes different command scheduling
decisions for different configurations. In general, the CROW-cache speedup
increases with the application’s memory intensity. Some memory-intensive
applications (e.g., _libq_ , _h264-dec_) have lower speedups than less-memory-
intensive applications because they exhibit high row buffer locality, which
reduces the benefits of CROW-cache. Applications with low MPKI ($<3$, not
plotted for brevity) achieve speedups below 5% due to limited memory activity,
but _no_ application experiences slow down.

Second, for most applications, CROW-1, with _only_ one copy row per subarray,
achieves most of the performance of configurations with more copy rows (on
average, 60% of the performance improvement of Ideal CROW-cache).

Third, the CROW table hit rate is very high for most of the applications. This
is indicative of high in-DRAM locality and translates directly into
performance improvement. On average, the hit rate for CROW-1/CROW-8/CROW-256
is 68.8%/85.3%/91.1%.

Fourth, the overhead of fully restoring rows evicted from the CROW table is
negligible (not plotted). For CROW-1, which has the highest overhead from
these evictions, restoring evicted rows accounts for only 0.6% of the total
activation operations.

#####  6.7.1.2 Multi-Core Performance.

Figure 6.6 shows the weighted speedup CROW-cache achieves with different CROW
substrate configurations for four-core workload groups. The bars show the
average speedup for the workloads in the corresponding workload group, and the
vertical lines show the maximum and minimum speedup among the workloads in the
group.

We observe that _all_ CROW-cache configurations achieve a higher speedup as
the memory intensity of the workloads increase. On average, CROW-8 provides
7.4% speedup for the workload group with four high-intensity workloads (i.e.,
_HHHH_), whereas it provides only 0.4% speedup for _LLLL_.

In contrast to single-core workloads, CROW-8 provides significantly better
speedup compared to CROW-1 on four-core configurations. This is because
simultaneously-running workloads are more likely to generate requests that
compete for the same subarray. As a result, the CROW table cannot achieve a
high hit rate with a single copy row per subarray. In most cases, CROW-8
performs close to Ideal CROW-cache with 100% hit rate, and requires only 1.6%
of the DRAM storage capacity for in-DRAM caching.

#####  6.7.1.3 DRAM Energy Consumption

We evaluate the total DRAM energy consumed during the execution of single-core
and four-core workloads. Figure 6.7 shows the average DRAM energy consumption
with CROW-cache normalized to the baseline. Although each ACT-c and ACT-t
command consumes more power than ACT, CROW-cache reduces the total DRAM energy
due to the improvement in execution time. On average, CROW-cache decreases
DRAM energy consumption by 8.2% and 6.9% on single- and four-core systems,
respectively.

?figurename? 6.6: CROW-cache speedup (160 four-core workloads).

?figurename? 6.7: DRAM energy consumption with CROW-cache.

#####  6.7.1.4 Comparison to TL-DRAM and SALP

We compare CROW-cache to two previous works that also enable in-DRAM caching
in different ways: TL-DRAM [2] and SALP [3].

TL-DRAM uses isolation transistors on each bitline to split a DRAM subarray
into a _far_ and a _near_ segment. When the isolation transistors are turned
off, the DRAM rows in the _near segment_ , which is composed of a small number
of rows close to the sense amplifiers, can be accessed with low tRCD and low
tRAS due to reduced parasitic capacitance and resistance on the short bitline.
In contrast, accessing the far segment requires a slightly higher tRCD and
tRAS than in conventional DRAM due to the additional parasitic capacitance and
resistance that the isolation transistor adds to the bitline. We extend our
circuit-level DRAM model to evaluate the reduction in DRAM latencies for
different far and near segment sizes. We use the notation TL-DRAM-$N_{r}$,
where $N_{r}$ specifies the number of rows in the near segment. TL-DRAM uses
the rows in the near segment as a cache by copying the most-recently accessed
rows in the far segment to the near segment. Thus, similar to our caching
mechanism, TL-DRAM requires an efficient in-DRAM row copy operation. We reuse
the ACT-c command that we implement for CROW-cache to perform the row copy in
TL-DRAM.

SALP [3] modifies the row decoder logic to enable parallelism among subarrays.
As opposed to a conventional DRAM bank where only a single row can be active
at a time, SALP enables the activation of multiple local row buffers
independently from each other to provide fast access to the most-recently-
activated row of each subarray. We evaluate the SALP-MASA mode, which
outperforms the other two modes that Kim et al. [3] propose. We evaluate SALP
with a different number of subarrays per bank, which we indicate as
SALP-$N_{s}$, where $N_{s}$ stands for the number of subarrays in a bank. For
SALP, we use both the timeout-based and the open-page (denoted as
SALP-$N_{s}$-O) row buffer management policies. The open-page policy keeps a
row open until a local row buffer conflict. Although this increases the
performance benefit of SALP by preventing a local row buffer from being
precharged before it is reused, SALP with open-page policy consumes more
energy since rows remain active for a longer time. Note that, in SALP, the in-
DRAM cache capacity changes with the number of subarrays in the DRAM chip as
each subarray can cache a single row in its local row buffer. To evaluate SALP
with higher cache capacity, we reduce the number of rows in each subarray by
increasing the number of subarrays in a bank, thus keeping the DRAM capacity
constant.

Figure 6.8 compares the performance, energy efficiency, and DRAM chip area
overhead of different configurations of CROW-cache, TL-DRAM [2], and SALP [3]
for single-core workloads. We draw four conclusions from the figure.

First, all SALP configurations using the open-page policy outperform CROW-
cache. However, SALP increases the DRAM energy consumption significantly as it
frequently keeps multiple local row buffers active, each of which consume
significant static power (as also shown in [3]). An idle LPDDR4 chip that has
only a single bank with an open row draws 10.9% more current ($I_{DD}3N$)
compared to the current ($I_{DD}2N$) when all banks are in closed state [188].
In SALP, the static power consumption is much higher than in conventional
DRAM, since multiple local row buffers per bank can be active at the same
time, whereas only one local row buffer per bank can be active in conventional
DRAM.

(a) Energy vs. speedup

(b) Chip area vs. speedup

?figurename? 6.8: CROW-cache vs. TL-DRAM [2] & SALP [3].

Second, increasing the number of subarrays in a bank increases the in-DRAM
cache capacity and, thus, the performance benefits of SALP. However, with the
open-page policy, SALP-256 has a 58.4% DRAM energy and 28.9% area overhead
over the baseline, which is much higher than the 27.3% DRAM energy and 0.6%
area overhead of SALP-128. SALP-512 has even larger DRAM energy (119.3%) and
area (84.5%) overheads (not plotted). In contrast, CROW-8 _reduces_ DRAM
energy (by 8.2%) at very low (0.48%) DRAM chip area overhead.

Third, TL-DRAM-8 provides a higher speedup of 13.8% compared to CROW-8, which
provides 7.1% speedup (while reserving the same DRAM storage capacity for
caching as TL-DRAM-8). This is because the latency reduction benefit of a very
small TL-DRAM near-segment, which comprises only one or eight rows, is higher
than the latency reduction benefit of CROW’s two-row activation. According to
our circuit-level simulations, a TL-DRAM near-segment with eight rows can be
accessed with a 73% reduction in tRCD and an 80% reduction in tRAS. However,
this comes at the cost of high DRAM chip area overhead, as we explain next.

Fourth, in TL-DRAM, the addition of an isolation transistor to each bitline
incurs high area overhead. As seen in Figure 6.8(b), TL-DRAM-8 incurs 6.9%
DRAM chip area overhead, whereas CROW-8 incurs _only_ 0.48% DRAM chip area
overhead.

We conclude that CROW enables a more practical and lower cost in-DRAM caching
mechanism than TL-DRAM and SALP.

#####  6.7.1.5 CROW-cache and Prefetcher.

We evaluate the performance benefits of CROW-cache on a system with a stride
prefetcher, which we implement based on the RPT prefetcher [402]. In Figure
6.9, we show the speedup that the prefetcher, CROW-cache, and the combination
of the prefetcher and CROW-cache achieve over the baseline, which does not
implement prefetching. For brevity, we only show results for a small number of
workloads, which we sampled to be representative of workloads where the
prefetcher provides different levels of effectiveness. We observe that, in
most cases, CROW-cache operates synergistically with the prefetcher, i.e.,
CROW-cache serves both read and prefetch requests with low latency, which
further improves average system performance by 5.7% over the prefetcher across
all single-core workloads.

?figurename? 6.9: CROW-cache and prefetching.

####  6.7.2 CROW-ref

Our weak row remapping scheme, CROW-ref, extends the refresh interval from
$64\text{\,}\mathrm{ms}$ to $128\text{\,}\mathrm{ms}$ of DRAM chip by
eliminating the small set of rows that have retention time below
$128\text{\,}\mathrm{ms}$. For our evaluations, we assume three weak rows for
each subarray, which is much more than expected, given data from real DRAM
devices [43, 122, 194].888Based on Equation 6.2 in section 6.3.2.1, the
probability of a having subarray with more than 3 weak rows across the entire
DRAM is $9.3\times 10^{-3}$. Extending the refresh interval provides
performance and energy efficiency benefits as fewer refresh operations occur
and they interfere less with application requests. CROW-ref does not incur any
additional overhead other than allocating a few of the copy rows that are
available with the CROW substrate. The rest of the copy rows can potentially
be used by other CROW-based mechanisms, e.g., CROW-cache.

Figure 6.10 shows CROW-ref’s average speedup and normalized DRAM energy
consumption for all single-core workloads and memory intensive four-core
workloads (_HHHH_) for four DRAM chip densities
($8163264\text{\,}\mathrm{Gbit}$). We observe that, for a futuristic
$64\text{\,}\mathrm{Gbit}$ DRAM chip, CROW-ref improves average performance by
7.1%/11.9% and reduces DRAM energy consumption by 17.2%/7.8% for
single-/multi-core workloads. The energy benefits are lower for four-core
workloads as DRAM accesses contribute a larger portion of the overall DRAM
energy compared to single-core workloads that inject relatively few requests
to the DRAM.

?figurename? 6.10: CROW-ref speedup and DRAM energy.

####  6.7.3 Combining CROW-cache and CROW-ref

In section 6.7.1, we note that not all applications effectively use all
available copy rows in CROW-cache since CROW-1 provides speedup close to CROW-
cache with more copy rows. Similarly, for CROW-ref, previous work shows that
there are only a small number (e.g., < 1000 on a $32\text{\,}\mathrm{GiB}$
DRAM [43]) of DRAM rows that must be refreshed at the lowest refresh
intervals, so it is very unlikely to have more than a few weak rows (or even
one) in a subarray. These two observations lead us to believe that the two
mechanisms, CROW-cache and CROW-ref, can be combined to operate
synergistically. We combine the two mechanisms such that CROW-cache utilizes
copy rows that remain available _after_ the weak row remapping of CROW-ref.
Combining the two mechanisms requires only a single additional bit per CROW
table entry to indicate whether a copy row is allocated for CROW-cache or
CROW-ref.

Figure LABEL:crow_fig:crow_cache_ref_results summarizes the performance and
energy efficiency benefits of CROW-cache, CROW-ref, and their combination for
an LLC capacity ranging from $512\text{\,}\mathrm{KiB}$ to
$32\text{\,}\mathrm{MiB}$ and $64\text{\,}\mathrm{Gbit}$ DRAM chip density.
The figure compares the benefits of the two mechanisms against a hypothetical
CROW-based mechanism that has an ideal CROW-cache with 100% hit rate and that
does not require any DRAM refresh. We make three observations from the figure.

First, the two mechanisms combined together provide higher performance and
energy efficiency than either mechanism alone. This is because each mechanism
improves a different aspect of DRAM. Using an $8\text{\,}\mathrm{MiB}$ LLC,
the combination of the two mechanisms provides a 20.0% performance improvement
and 22.3% DRAM energy reduction for four-core workloads. The performance and
energy benefits of the combined CROW-cache/CROW-ref mechanisms is larger than
the the _sum_ of the benefits of each individual mechanism. This is because by
eliminating many refresh operations, CROW-ref decreases the interference of
refresh operations with DRAM access operations. This helps CROW-cache, as a
row that would have been closed by a refresh operation may now stay open, and
no longer needs to be reactivated. As a result, CROW-cache can use its limited
copy rows to help accelerate the activation of other rows than the ones that
would have been closed by refresh.

Second, CROW-cache and CROW-ref significantly improve performance and reduce
DRAM energy for _all_ LLC capacities. For four-core workloads, we observe
22.4%/20.0%/15.7% performance improvement and 22.0%/22.3%/22.8% DRAM energy
reduction for $1832\text{\,}\mathrm{MiB}$ LLC capacity.

Third, the combination of CROW-cache and CROW-ref achieves performance and
DRAM energy improvements close to the ideal mechanism (100% hit rate CROW-
cache and no refresh). Using an $8\text{\,}\mathrm{MiB}$ LLC, the combination
of the two mechanisms achieves 71% of the performance and 99%999The
combination of CROW-cache and CROW-ref almost reaches the energy reduction of
the ideal mechanism because the ideal mechanism _always_ hits in the CROW
table, and thus _always_ uses an ACT-t command to activate a row (which
consumes more energy than a regular ACT). of the DRAM energy improvements of
the ideal mechanism.

We conclude that CROW is a flexible substrate that enables multiple different
mechanisms to operate simultaneously, thereby improving both DRAM performance
and energy efficiency.

###  6.8 Related Work

To our knowledge, we are first to propose a flexible and low-cost DRAM
substrate that enables _multiple_ mechanisms for improving DRAM performance,
energy efficiency, and reliability. We briefly discuss closely related prior
work that propose in-DRAM caching, latency reduction, refresh reduction, and
RowHammer protection mechanisms.

In-DRAM Caching Mechanisms. In section 6.7.1.4, we qualitatively and
quantitatively compare CROW to the two most-closely related prior works, TL-
DRAM [2] and SALP [3], which also propose in-DRAM caching. We show that CROW-
cache is lower-cost to implement and consumes less DRAM energy compared to the
two mechanisms. Since TL-DRAM and SALP exploit _different_ observations to
enable low-latency regions in DRAM, CROW-cache can be implemented in
combination with both TL-DRAM and SALP to further improve system performance.

Chang et al. [39] propose LISA to enable efficient data movement between two
subarrays. They also propose _LISA-VILLA_ , which changes the bank
architecture to include small (but fast) subarrays [403] to enable dynamic in-
DRAM caching. FIGARO [133] proposes a DRAM architecture that enables fine-
grained data relocation within a DRAM bank. LISA and FIGARO 1) require more
changes to the DRAM chip compared to CROW-cache and 2) are complementary to
CROW-cache as a copy row can be implemented in the subarrays of both designs.

Hidaka et al. [185] propose to add SRAM memory cells inside a DRAM chip to
utilize as a cache. Implementing SRAM-based memory in DRAM incurs very high
area overhead (e.g., 38.8% of the DRAM chip area for $64\text{\,}\mathrm{KiB}$
SRAM cache as shown in [3, 2]). Prior works [404, 405, 186] that propose
implementing multiple row buffers (that can be used as cache) also suffer from
high area overhead. CROW-cache can be used in combination with all these prior
mechanisms to reduce the latency of fetching data from a DRAM row to the in-
DRAM cache.

Multiple Clone Row (MCR-DRAM) [179] is based on an idea that is similar to
simultaneously activating multiple duplicate rows. The key idea is to
dynamically configure a DRAM bank into different access-latency regions, where
a region stores data in a single row or duplicates data into multiple rows.
MCR-DRAM does _not_ propose a hardware-managed in-DRAM cache as it requires
support from the operating system (or the application) to manage 1) the size
of each region and 2) data movement between different regions. However, both
operations are difficult to perform for the operating system due to memory
fragmentation, which complicates dynamic resizing of a region and determining
which data would benefit from a low-latency region. CLR-DRAM [15] is a new
DRAM architecture that enables dynamically coupling two adjacent DRAM cells to
operate as a single cell for low latency access. Similar to MCR-DRAM, CLR-DRAM
requires software support for determining which data to place in low-latency
regions. Compared to MCR-DRAM and CLR-DRAM, CROW is more practical to
implement, as it is hardware managed and thus completely transparent to the
software.

Mitigating Refresh Overhead. Many prior works tackle the refresh overhead
problem in DRAM. RAIDR [43] proposes to bin DRAM rows based on their retention
times and perform refresh operations at a different rate on each bin. Riho et
al. [192] use a technique based on simultaneous multiple-row activation.
However, their mechanism is specifically designed for optimizing only refresh
operations, and thus it is not as flexible as the CROW substrate, which
provides copy rows that enable multiple orthogonal mechanisms. Other prior
works [46, 49, 50, 51, 53, 54, 55, 189, 162, 367, 302, 190, 57, 45, 58, 193,
59, 41, 66, 68, 69, 191, 122, 67] propose various techniques to optimize
refresh operations. These works are specifically designed for _only_
mitigating the refresh overhead. In contrast, CROW provides a versatile
substrate that can simultaneously reduce the refresh overhead, reduce latency,
and improve reliability.

Reducing DRAM Latency. ChargeCache [17] reduces the average DRAM latency based
on the observation that recently-precharged rows, which are fully restored,
can be accessed faster compared to rows that have leaked some charge and are
about to be refreshed soon. Note that ChargeCache enables low-latency access
when DRAM rows are repeatedly activated in very short intervals, e.g.,
$1\text{\,}\mathrm{ms}$. In contrast, a copy row in CROW-cache enables low-
latency access to a regular row for an indefinite amount of time (until the
row is evicted from CROW table). Thus, CROW-cache captures higher amount of
in-DRAM locality. CROW-cache is also orthogonal to ChargeCache and the two
techniques can be implemented together.

Recent studies [11, 159, 102, 137, 10] propose mechanisms to reduce the DRAM
access latency by reducing the margins present in timing parameters when
operating under appropriate conditions (e.g., low temperature). Other works
[186, 9, 132, 129, 41, 99, 126, 184, 11] propose different methods to reduce
DRAM latency. These works are orthogonal to CROW, and they can be implemented
along with our mechanism to further reduce DRAM latency.

###  6.9 Summary

We propose CROW, a low-cost substrate that partitions each DRAM subarray into
two regions (regular rows and copy rows) and enables independent control over
the rows in each region. We leverage CROW to design two new mechanisms, CROW-
cache and CROW-ref, that improve DRAM performance and energy-efficiency. CROW-
cache uses a copy row to duplicate a regular row and simultaneously activates
a regular row together with its duplicated copy row to reduce the DRAM
activation latency (by 38% in our experiments). CROW-ref remaps retention-weak
regular rows to strong copy rows, thereby reducing the DRAM refresh rate.
CROW’s flexibility allows us to simultaneously employ both CROW-cache and
CROW-ref to provide 20.0% speedup and 22.3% DRAM energy savings over
conventional DRAM. We conclude that CROW is a flexible DRAM substrate that
enables a wide range of mechanisms to improve performance, energy efficiency,
and reliability. We freely open source CROW [123] to enable future work in
this area. We hope future work exploits CROW to devise more use cases that can
take advantage of its low-cost and versatile substrate.

##  ?chaptername? 7 Conclusions and Future Directions

In this dissertation, we demonstrate that new understanding developed based on
rigorous characterization of real DRAM chips leads to new mechanisms for
improving DRAM performance, energy consumption, reliability, and security. We
enable detailed characterization of modern DRAM chips by designing and
prototyping a flexible FPGA-based DRAM testing infrastructure (the first of
its kind), which we use to develop new insights and new mechanisms for
improving system performance, efficiency, reliability, and security.

First, in chapter 3, we introduce _SoftMC_ , the first publicly available
FPGA-based DRAM testing infrastructure. SoftMC provides a flexible and easy-
to-use software interface that enables testing any combination of any standard
DRAM operations. We demonstrate the capability, flexibility, and programming
ease of SoftMC via two example use cases. Our experimental analyses
demonstrate the effectiveness of SoftMC as a new tool to _(i)_ perform
detailed characterization of various DRAM parameters (e.g., refresh interval
and access latency) as well as the relationships between them, and _(ii)_ test
the expected effects of existing or new mechanisms (e.g., whether or not
highly-charged cells can be accessed faster in existing DRAM chips). We
believe and hope that SoftMC, with its flexibility and ease of use, can enable
many other studies, ideas and methodologies in the design of future memory
systems, by making memory control and characterization easily accessible to a
wide range of software and hardware developers. In fact, the improved DDR4
version of SoftMC, called DRAM Bender, is released as an open-source paper
[406] and tool [407] to enable such further studies.

Second, in chapter 4, we use SoftMC to develop U-TRR, a new methodology for
uncovering the operation of in-DRAM Target Row Refresh (TRR) mechanisms
implemented in existing DRAM chips as a protection against the RowHammer
vulnerability. Using U-TRR, we 1) provide insights into the inner workings of
existing proprietary and undocumented TRR mechanisms and 2) develop custom
DRAM access patterns to efficiently circumvent TRR in DDR4 DRAM modules from
three major vendors. We demonstrate that existing in-DRAM TRR mechanisms are
not secure against RowHammer and can be easily circumvented using the new
RowHammer access patterns that we develop based on new insights gained via
U-TRR experiments. We believe and hope that U-TRR will inspire future research
and lead to the development of both new RowHammer attacks and secure RowHammer
protection mechanisms.

Third, in chapter 5, we propose Self-Managing DRAM (SMD), a new DRAM interface
and architecture that enables easy adoption of existing and new DRAM
maintenance operations that improve DRAM performance, efficiency, and
robustness. To achieve this, SMD sets the memory controller free from managing
DRAM maintenance operations with minimal changes to the existing DRAM chips
and memory controllers. Using SMD, we implement efficient maintenance
mechanisms for DRAM refresh, RowHammer protection, and memory scrubbing. We
show that these mechanisms altogether enable a higher performance, more energy
efficient and at the same time more reliable and secure DRAM system. We
believe and hope that SMD will enable practical and easier adoption of
innovative ideas in DRAM design as well as development of efficient in-DRAM
maintenance mechanisms.

Finally, in chapter 6, we propose Copy-Row DRAM (CROW), a low-cost substrate
that enables mechanisms for improving DRAM latency and energy efficiency. CROW
partitions a DRAM subarray into two regions (regular rows and s) and enables
independent control over the rows in each region. We use CROW to design two
new mechanisms. CROW-cache reduces DRAM access latency by duplicating a
regular row’s data to a and simultaneously activating both rows to reduce row
activation latency by 38%. CROW-ref reduces DRAM refresh rate, and thus the
performance and energy overhead of DRAM refresh overhead, by remapping
retention-weak regular rows to strong s. We simultaneously employ both CROW-
cache and CROW-ref to provide 20.0% speedup and 22.3% DRAM energy savings over
conventional DRAM. We conclude that CROW is a flexible DRAM substrate that
enables a wide range of mechanisms to improve performance, energy efficiency,
and reliability. We believe and hope that future work will take advantage of
CROW to devise more mechanisms that improve various aspects of DRAM.

In conclusion, this dissertation shows that understanding DRAM characteristics
and operation via rigorous experiments using real DRAM chips is critical for
developing new mechanisms for improving DRAM performance, efficiency,
reliability, and security. We believe and hope that the DRAM characterization
infrastructure, understanding, and techniques we develop in this dissertation
will encourage more innovation in the design of future memory systems.

###  7.1 Future Research Directions

SoftMC provides a practical infrastructure for studying the reliability and
latency characteristics of real DRAM chips. Our follow-up works, U-TRR, SMD,
and CROW show that new observations and insights on DRAM reliability/latency
characteristics and understanding of intrinsic DRAM operation can be exploited
to improve performance, efficiency, reliability, and security aspects of DRAM.
We believe our works will enable new research directions. In this section, we
discuss several such potential future research directions.

####  7.1.1 Deeper DRAM Characterization

Many works [29, 99, 100, 101, 102, 69, 104, 10, 71, 70, 105, 106, 103, 75,
107, 108, 109, 111, 114, 113] use SoftMC to perform experiments using real
DRAM chips. Going forward, we expect SoftMC to continue having a key role in
enabling future research on various aspects of DRAM and potentially other
memory technologies. We discuss four such example research directions.

First, continuous DRAM technology node scaling leads to increasing reliability
and latency issues in DRAM [5, 61, 269, 270, 43, 42, 65, 67, 68, 66, 69, 74,
70, 71]. Thus, it is essential to continuously study the characteristics of
new DRAM generations to develop mechanisms for improving DRAM reliability and
performance of future DRAM chips. Our SoftMC infrastructure can help
researchers answer various questions to this end: How are the DRAM cell
characteristics, reliability, and latency changing with different generations
of technology nodes? Do all DRAM operations and cells get affected by scaling
at the same rate? Which DRAM operations are getting worse? What are the future
trends in the RowHammer vulnerability of DRAM? Do other types of disturbance
errors manifest in new technology node generations?

Second, the existing research on the impact of aging on DRAM reliability and
latency characteristics is very limited [196, 197]. A recent work [111]
performs experiments with 3 DDR3 modules to study the effects of accelerated
aging on the retention behavior of the DRAM cells. In their analysis, the
authors observe increase in DRAM retention errors after 8 and 16 hours of
accelerated aging. Although this work presents valuable preliminary
experimental data, we need to perform thorough experiments across a much
larger set of DRAM chips across different generations. Additionally, we need
to study how aging effects errors rates with reduced DRAM timing parameters.
Overall, the causes, characteristics, and impact of aging have remained
largely unstudied. Using SoftMC, it is possible to devise controller
experiments to analyze and characterize aging over very long time periods.
SoftMC can help researchers answer questions such as: How prevalent are aging-
related failures? What types of usage accelerate aging? Can operation under
high temperature and voltage accelerate aging and if so, to what degree? How
can we design architectural or system-level techniques that can slow down the
aging process?

Third, prior works show that the failure rate of DRAM modules in large data
centers is significant, largely affecting the cost and downtime in data
centers [197, 272, 196, 302]. Unfortunately, there is no study that analyzes
DRAM modules that have failed in the field to determine the common causes of
failure. Our SoftMC infrastructure can test faulty DRAM modules and help
answer various research questions: What are the dominant types of DRAM
failures at runtime? Are failures correlated to any location or specific
structure in DRAM? Do all chips from the same generation exhibit the same
failure characteristics?

Fourth, memories that can operate at extremely low temperatures (e.g.,
$77\text{\,}\mathrm{K}$ and below) are critical for the realization of
cryogenic processors, e.g., quantum computers [408, 409] and superconducting
processors [409, 410]. Although recent works (e.g., [409, 411, 408, 412, 413,
414]) analyze the retention and reliability properties of DRAM operating at
cryogenic temperatures, we need to conduct more rigorous experimental studies
using real DRAM chips to fully understand the reliability and latency
characteristics of modern DRAM operating at very low temperatures. SoftMC can
help researchers answer research questions like: How reliably can a DRAM chip
operate at very low temperatures? How do different DRAM designs perform under
cryogenic temperatures? How can we improve DRAM reliability at very low
temperatures? How are different DRAM latencies affected by very low
temperatures and how can we improve DRAM access latency under such operating
conditions?

####  7.1.2 Extending SoftMC to Support Other DRAM Standards and Memory
Technologies

SoftMC can be used out-of-the-box to test DDR3 chips, and the new version of
SoftMC, called DRAM Bender [406, 407] can be used to test DDR4 chips. However,
it can be extended to support other DRAM interfaces such as different
generations of HBM [415], GDDR [175], and LPDDR [118]. Besides enabling tests
on a wide range of DRAM chips, such a design also makes the scope of chips
that SoftMC can test go beyond just DRAM. With the emergence of byte-
addressable Non-Volatile Memories (NVM) (e.g., PCM [303, 304, 305, 416, 417,
418], STT-RAM [306, 307, 419, 420, 421, 422, 311, 423], ReRAM [308, 309, 424,
425]) in DDR-compatible form factors [310, 311, 426, 427], it is critical to
characterize and analyze such NVM chips in order to understand and exploit
them as well as improve their design. We believe that SoftMC can be seamlessly
used to characterize these chips, and can help enable future mechanisms for
NVM.

####  7.1.3 Improving RowHammer Attacks and Defenses

Prior works [78, 70, 71, 73, 134, 100, 369, 75, 97, 94, 95, 96] show that DRAM
technology node scaling leads to 1) reduction in minimum row activation count
($HC_{first}$) for causing a RowHammer bit flip and 2) increase in RowHammer
blast radius (i.e., the range of neighbor rows on which hammering an aggressor
row causes RowHammer bit flips). To efficiently protect future DRAM chips with
increasing vulnerability to RowHammer, the performance, energy, and area
overheads of a RowHammer mitigation mechanism must be small even for extremely
low $HC_{first}$. The SoftMC infrastructure and the U-TRR methodology can be
used to study the efficiency and security guarantees of future RowHammer
protection mechanisms.

As RowHammer defenses improve, studying the security guarantees of such
defenses implemented in commodity DRAM chips will remain as a critical
challenge. Specifically, there is a need for discovering new access patterns
that can circumvent the RowHammer protection mechanisms and exploring the
effects of various factors (e.g., data pattern dependence, operating
temperature and voltage) on the RowHammer vulnerability level of future DRAM
chips will remain as an important research direction, especially for
undocumented RowHammer protection mechanisms that rely on security-through-
obscurity but also for well-documented mechanisms that are implemented in real
hardware.

In U-TRR, we show that our carefully crafted access patterns can cause too
many bit flips in a codeword for simple ECC to detect and correct. However, on
systems equipped with ECC, mounting a RowHammer attack that could manipulate
sensitive data by flipping bits in DRAM is challenging. This is because an
attacker may unintentionally cause a detectable error while searching for a
DRAM location and access/data pattern that would result in undetectable number
of errors. When a detectable (but uncorrectable) error occurs, a system may
hang, restart, or crash, leading to denial-of-service and disruption of the
RowHammer attack. Developing a methodology that overcomes this challenge and
efficiently compromises a system despite of the presence of ECC is a promising
research direction.

####  7.1.4 New DRAM Maintenance Mechanisms

In chapter 5, we show the limitations of existing DRAM interfaces impose
against designing efficient DRAM maintenance mechanisms. Using Self-Managing
DRAM (SMD), we demonstrate how to design efficient in-DRAM maintenance
mechanisms for periodic refresh, RowHammer protection, and memory scrubbing.
As SMD eases the adoption of new DRAM maintenance mechanisms, researchers can
further optimize these maintenance mechanisms or propose new mechanisms for
the same purposes or for other maintenance operations. We believe SMD can also
be used to design maintenance mechanisms for other purposes. For example,
using SMD, one can implement various profiling-guided mechanisms for improving
DRAM performance and energy efficiency by exploiting design-induced variation
across DRAM cells [126, 184]. A maintenance mechanism can automatically
identify low-latency DRAM rows that can be reliably accesses with lowered
timing parameters. As in other SMD-based maintenance mechanisms, SMD can
prevent the memory controller from accessing a region that is being profiled.
By conveying the profiling results to the memory controller and the operating
system, such low-latency rows can be used to place frequently-accessed data.

SMD can also ease managing the coordination between a near-/in-DRAM processing
engine [101, 129, 128, 106, 208, 140, 130, 428, 429, 430, 27, 431, 432, 433,
434, 114, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447,
448, 449, 450, 34] and a memory controller. The ability to reject row
activations, which SMD provides, can help resolve access conflicts between a
near-/in-DRAM processing engine and a memory controller in the same system. To
do so, SMD can treat the near-/in-DRAM processing engine as a maintenance
mechanism. The engine can use SMD to lock a DRAM region that it will operate
on. Because SMD does not allow the memory controller to activate a row in a
locked region, only the near-/in-DRAM processing engine will have access to
the locked region until it completes the processing and releases the region.

####  7.1.5 Enabling in-DRAM Data Movement

The CROW substrate we presented in chapter 6 exploits simultaneous multiple
row activation and in-subarray data movement to develop mechanisms for in-DRAM
caching and reducing DRAM refresh overhead. CROW can enable various other
mechanisms, such as the RowHammer mitigation mechanism we discuss in section
6.3.3 and reduce the overheads of new RowHammer mitigation proposals, e.g.,
[158, 95, 96]. We believe even wider range of mechanisms for improving DRAM
performance, energy efficiency, and reliability can be enabled with either
CROW or similar simple changes in the DRAM architecture.

###  7.2 Final Concluding Remarks

In this dissertation, we demonstrate the importance of rigorously
understanding DRAM characteristics and operation of existing DRAM chips on
developing new mechanisms for improving DRAM performance, energy efficiency,
reliability, and security. We hope that the contributions in this dissertation
enable future research in many area in future DRAM and memory systems,
including understanding the characteristics of current and future memory chips
and systems, improving RowHammer attacks and defenses, adopting efficient DRAM
maintenance mechanisms, exploiting in-DRAM data movement, and many other
areas.

##  ?appendixname? A Other Works of the Author

During my doctoral studies, I had the opportunity to contribute on several
different areas with researchers from ETH Zürich and other institutions. In
this chapter, I acknowledge these works.

Besides the works presented in this dissertation, I worked on many other DRAM
projects. In collaboration with Jeremie S. Kim, we developed 1) D-RaNGE [102],
a mechanism for generating high-throughput true random numbers using DRAM, 2)
DRAM Latency PUF [137], a new class of fast and reliable DRAM-based Physical
Unclonable Functions (PUFs), and 3) Solar-DRAM [184], a mechanism for reducing
DRAM access latency by exploiting variation in bitlines. In collaboration with
Minesh Patel, we analyzed modern DRAM chips with on-die ECC and developed a
methodology to extract pre-correction error characteristics of DRAM chips
[135, 451] and a methodology to determine the full DRAM on-die ECC function
[141, 452]. In collaboration with Kevin Chang, we characterized the latency
and reliability characteristics of DRAM chips to understand their intrinsic
latency variation [10] and operation under reduced voltage [99].

I contributed to several other DRAM projects. CLR-DRAM [15, 453] is a new DRAM
architecture that enables dynamic capacity-latency trade-off by enabling a
DRAM row to switch between max-capacity and high-performance modes. In CAL
[183], we proposed a mechanism that reduces DRAM charge restoration time, and
thus reduces DRAM access latency, by partially restoring DRAM cell charge when
a DRAM row is predicted to be soon re-activated. VRL-DRAM [48] proposes a
mechanism that reduces DRAM refresh overhead by partially refreshing a row
when it can correctly retain its data until the next time it gets refreshed.
In Refresh Triggered Computation [47], we propose a technique for eliminating
redundant refresh operations by exploiting the regular memory access patterns
of convolutional neural network applications. In [104], we characterize the
power consumption of real DRAM chips and develop an accurate DRAM power model
based on the observations we develop from our characterization study. DR-
STRaNGe [139] is an end-to-end design for enabling DRAM-based true random
number generation mechanisms. HIRA [113] reduces the DRAM refresh latency in
off-the-shelf DRAM chips by exploiting back-to-back activation of two rows to
allow refreshing one of the activated rows while accessing the other one. DRAM
Bender [406, 407] is a new DRAM characterization infrastructure that extends
SoftMC by enabling support for DDR4 chips and introducing a new programming
model that overcomes the program size limitation of SoftMC in developing DRAM
experiments. in Sectored DRAM [454], we propose a low-overhead DRAM substrate
that enables fine-grained DRAM row activation and access.

I contributed to works that enable computation and other functionality within
DRAM chips. Ambit [129] proposes an in-DRAM accelerator for performing bulk
bitwise operations. CODIC [106] enables fine-grained control over previously
fixed internal DRAM timings and develops a new DRAM-based PUF and a mechanism
for preventing DRAM cold boot attacks. In [208, 455, 456], we develop PiDRAM,
a flexible open-source framework that enables system integration of mechanisms
that perform processing using memory.

I also worked on projects that analyze the RowHammer vulnerability of DRAM and
existing defense mechanisms against it. In [70] and [71], we characterize the
RowHammer vulnerability of many DRAM chips across different vendors and
technology nodes and study the sensitivity of RowHammer to factors such as
operating temperature and aggressor row active time. TRRespass [100] shows
that many modern DRAM chips equipped with a Target Row Refresh (TRR)
mechanism, which is advertised as complete solution to the RowHammer problem,
are in fact vulnerable to access patterns that utilize many aggressor rows.
BlockHammer [78, 457] is a low-cost and easy-to-adopt RowHammer mitigation
mechanism that efficiently tracks aggressor rows using Bloom Filters and
throttles future activations to the aggressor row until its neighbors are
refreshed. In [75], we characterize the effects of reducing DRAM voltage on
the RowHammer vulnerability.

I also contributed to works outside my main research area. In CoNDA [458] (and
an earlier version [459]), we propose an efficient cache coherence mechanism
for systems equipped with a near-data accelerator. MetaSys [460, 461] is an
open-source FPGA-based framework for rapidly evaluating new cross-layer
techniques. In GRIM-Filter [462], we propose a DNA read mapping seed filtering
algorithm that is optimized to exploit 3D-stacked memory systems to perform
seed filtering in memory. GateKeeper [463, 464] is a highly-accurate hardware
accelerator for accelerating the pre-alignment step of short DNA read mapping.
Shouji [465, 466] is a highly-parallel and accurate DNA read mapping pre-
alignment filter that significantly reduces the need for expensive dynamic
programming algorithms.

##  ?refname?

  * [1] H. Hassan, N. Vijaykumar, S. Khan, S. Ghose, K. Chang, G. Pekhimenko, D. Lee, O. Ergin, and O. Mutlu, “SoftMC: A Flexible and Practical Open-Source Infrastructure for Enabling Experimental DRAM Studies,” in _HPCA_ , 2017\.
  * [2] D. Lee, Y. Kim, V. Seshadri, J. Liu, L. Subramanian, and O. Mutlu, “Tiered-Latency DRAM: A Low Latency and Low Cost DRAM Architecture,” in _HPCA_ , 2013.
  * [3] Y. Kim, V. Seshadri, D. Lee, J. Liu, and O. Mutlu, “A Case for Exploiting Subarray-Level Parallelism (SALP) in DRAM,” in _ISCA_ , 2012.
  * [4] R. H. Dennard, “Field-Effect Transistor Memory,” 1968, US Patent 3,387,286.
  * [5] O. Mutlu, “Memory Scaling: A Systems Architecture Perspective,” in _IMW_ , 2013.
  * [6] O. Mutlu and L. Subramanian, “Research Problems and Opportunities in Memory Systems,” in _SUPERFRI_ , 2014.
  * [7] JEDEC, _DDR3 SDRAM Specification_ , 2008.
  * [8] JEDEC, _DDR4 SDRAM Specification_ , 2012.
  * [9] Y. H. Son, O. Seongil, Y. Ro, J. W. Lee, and J. H. Ahn, “Reducing Memory Access Latency with Asymmetric DRAM Bank Organizations,” in _ISCA_ , 2013\.
  * [10] K. K. Chang, A. Kashyap, H. Hassan, S. Ghose, K. Hsieh, D. Lee, T. Li, G. Pekhimenko, S. Khan, and O. Mutlu, “Understanding Latency Variation in Modern DRAM Chips: Experimental Characterization, Analysis, and Optimization,” in _SIGMETRICS_ , 2016.
  * [11] D. Lee, Y. Kim, G. Pekhimenko, S. Khan, V. Seshadri, K. Chang, and O. Mutlu, “Adaptive-Latency DRAM: Optimizing DRAM Timing for the Common-Case,” in _HPCA_ , 2015.
  * [12] D. Lee, “Reducing DRAM Latency at Low Cost by Exploiting Heterogeneity,” Ph.D. dissertation, Carnegie Mellon University, 2016.
  * [13] M. Patel, “Enabling Effective Error Mitigation in Modern Memory Chips that Use On-Die Error-Correcting Codes,” Ph.D. dissertation, ETH Zürich, 2022.
  * [14] M. Patel, T. Shahroodi, A. Manglik, A. G. Yaglikci, A. Olgun, H. Luo, and O. Mutlu, “A Case for Transparent Reliability in DRAM Systems,” _arXiv preprint arXiv:2204.10378_ , 2022.
  * [15] H. Luo, T. Shahroodi, H. Hassan, M. Patel, A. Giray Yağlıkçı, L. Orosa, J. Park, and O. Mutlu, “CLR-DRAM: A Low-Cost DRAM Architecture Enabling Dynamic Capacity-Latency Trade-Off,” in _ISCA_ , 2020.
  * [16] S. Borkar and A. A. Chien, “The Future of Microprocessors,” in _CACM_ , 2011\.
  * [17] H. Hassan, G. Pekhimenko, N. Vijaykumar, V. Seshadri, D. Lee, O. Ergin, and O. Mutlu, “ChargeCache: Reducing DRAM Latency by Exploiting Row Access Locality,” in _HPCA_ , 2016.
  * [18] M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A. D. Popescu, A. Ailamaki, and B. Falsafi, “Clearing The Clouds: A Study Of Emerging Scale-Out Workloads On Modern Hardware,” _ASPLOS_ , 2012.
  * [19] “Moby: A mobile benchmark suite for architectural simulators.”
  * [20] A. Gutierrez, R. G. Dreslinski, T. F. Wenisch, T. Mudge, A. Saidi, C. Emmons, and N. Paver, “Full-System Analysis And Characterization Of Interactive Smartphone Applications,” in _IISWC_ , 2011.
  * [21] Y. Zhu, D. Richins, M. Halpern, and V. J. Reddi, “Microarchitectural Implications Of Event-Driven Server-Side Web Applications,” in _MICRO_ , 2015.
  * [22] J. Hestness, S. W. Keckler, and D. A. Wood, “A Comparative Analysis Of Microarchitecture Effects On CPU and GPU Memory System Behavior,” in _IISWC_ , 2014.
  * [23] S. Ghose, T. Li, N. Hajinazar, D. S. Cali, and O. Mutlu, “Demystifying Complex Workload-DRAM Interactions: An Experimental Study,” _SIGMETRICS_ , 2019.
  * [24] O. Mutlu, J. Stark, C. Wilkerson, and Y. N. Patt, “Runahead Execution: An Alternative to Very Large Instruction Windows for Out-of-Order Processors,” in _HPCA_ , 2003.
  * [25] R. Bera, A. V. Nori, O. Mutlu, and S. Subramoney, “DSPatch: Dual Spatial Pattern Prefetcher,” in _MICRO_ , 2019.
  * [26] A. Boroumand, S. Ghose, Y. Kim, R. Ausavarungnirun, E. Shiu, R. Thakur, D. Kim, A. Kuusela, A. Knies, P. Ranganathan, and O. Mutlu, “Google Workloads for Consumer Devices: Mitigating Data Movement Bottlenecks,” in _ASPLOS_ , 2018\.
  * [27] S. Ghose, A. Boroumand, J. S. Kim, J. Gómez-Luna, and O. Mutlu, “Processing-In-Memory: A Workload-Driven Perspective,” _IBM Journal of Research and Development_ , 2019.
  * [28] S. Kanev, J. P. Darago, K. Hazelwood, P. Ranganathan, T. Moseley, G.-Y. Wei, and D. Brooks, “Profiling a Warehouse-Scale Computer,” in _ISCA_ , 2015\.
  * [29] S. Koppula, L. Orosa, A. G. Yağlıkçı, R. Azizi, T. Shahroodi, K. Kanellopoulos, and O. Mutlu, “EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM,” in _MICRO_ , 2019.
  * [30] X. Liu, D. Roberts, R. Ausavarungnirun, O. Mutlu, and J. Zhao, “Binary Star: Coordinated Reliability in Heterogeneous Memory Systems for High Performance and Scalability,” in _MICRO_ , 2019.
  * [31] M. V. Wilkes, “The Memory Gap and the Future of High Performance Memories,” _ACM SIGARCH Computer Architecture News_ , 2001.
  * [32] W. A. Wulf and S. A. McKee, “Hitting the Memory Wall: Implications of the Obvious,” _ACM SIGARCH Computer Architecture Mews_ , 1995.
  * [33] G. F. Oliveira, J. Gómez-Luna, L. Orosa, S. Ghose, N. Vijaykumar, I. Fernandez, M. Sadrosadati, and O. Mutlu, “DAMOV: A New Methodology and Benchmark Suite for Evaluating Data Movement Bottlenecks,” _IEEE Access_ , 2021.
  * [34] A. Boroumand, S. Ghose, B. Akin, R. Narayanaswami, G. F. Oliveira, X. Ma, E. Shiu, and O. Mutlu, “Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks,” in _PACT_ , 2021.
  * [35] A. Boroumand, S. Ghose, G. F. Oliveira, and O. Mutlu, “Polynesia: Enabling High-Performance and Energy-Efficient Hybrid Transactional/Analytical Databases with Hardware/Software Co-Design,” in _ICDE_ , 2022.
  * [36] A. Boroumand, “Practical Mechanisms for Reducing Processor-Memory Data Movement in Modern Workloads,” Ph.D. dissertation, Carnegie Mellon University, 2020.
  * [37] _RLDRAM Memory_ , Micron Technology, 2021, https://www.micron.com/products/dram/rldram-memory.
  * [38] Y. Sato, T. Suzuki, T. Aikawa, S. Fujioka, W. Fujieda, H. Kobayashi, H. Ikeda, T. Nagasawa, A. Funyu, Y. Fuji, K. Kawasaki, M. Yamazaki, and M. Taguchi, “Fast Cycle RAM (FCRAM); A 20-ns Random Row Access, Pipe-Lined Operating DRAM,” in _VLSIC_ , 1998.
  * [39] K. K. Chang, P. J. Nair, D. Lee, S. Ghose, M. K. Qureshi, and O. Mutlu, “Low-Cost Inter-Linked Subarrays (LISA): Enabling Fast Inter-Subarray Data Movement in DRAM,” in _HPCA_ , 2016.
  * [40] ITRS, “International Technology Roadmap for Semiconductors Executive Summary,” 2013, http://www.itrs2.net/2013-itrs.html.
  * [41] K. K. Chang, D. Lee, Z. Chishti, A. R. Alameldeen, C. Wilkerson, Y. Kim, and O. Mutlu, “Improving DRAM Performance by Parallelizing Refreshes with Accesses,” in _HPCA_ , 2014.
  * [42] U. Kang, H.-s. Yu, C. Park, H. Zheng, J. Halbert, K. Bains, S. Jang, and J. S. Choi, “Co-Architecting Controllers and DRAM to Enhance DRAM Process Scaling,” in _The Memory Forum_ , 2014.
  * [43] J. Liu, B. Jaiyen, R. Veras, and O. Mutlu, “RAIDR: Retention-Aware Intelligent DRAM Refresh,” in _ISCA_ , 2012.
  * [44] JEDEC, “DDR5 SDRAM Specification - JESD79-5A,” 2021.
  * [45] J. Mukundan, H. Hunter, K.-h. Kim, J. Stuecheli, and J. F. Martínez, “Understanding and Mitigating Refresh Overheads in High-Density DDR4 DRAM Systems,” in _ISCA_ , 2013.
  * [46] P. J. Nair, C.-C. Chou, and M. K. Qureshi, “Refresh Pausing in DRAM Memory Systems,” in _TACO_ , 2014.
  * [47] S. M. Jafri, H. Hassan, A. Hemani, and O. Mutlu, “Refresh Triggered Computation: Improving the Energy Efficiency of Convolutional Neural Network Accelerators,” _TACO_ , 2020.
  * [48] A. Das, H. Hassan, and O. Mutlu, “VRL-DRAM: Improving DRAM Performance Via Variable Refresh Latency,” in _DAC_ , 2018.
  * [49] M. K. Qureshi, D.-H. Kim, S. Khan, P. J. Nair, and O. Mutlu, “AVATAR: A Variable-Retention-Time (VRT) Aware Refresh for DRAM Systems,” in _DSN_ , 2015.
  * [50] S. Baek, S. Cho, and R. Melhem, “Refresh Now and Then,” in _TC_ , 2014.
  * [51] I. Bhati, Z. Chishti, and B. Jacob, “Coordinated Refresh: Energy Efficient Techniques for DRAM Refresh Scheduling,” in _ISPLED_ , 2013.
  * [52] I. Bhati, Z. Chishti, S.-L. Lu, and B. Jacob, “Flexible Auto-Refresh: Enabling Scalable and Energy-Efficient DRAM Refresh Reductions,” in _ISCA_ , 2015.
  * [53] Z. Cui, S. A. McKee, Z. Zha, Y. Bao, and M. Chen, “DTail: A Flexible Approach to DRAM Refresh Management,” in _SC_ , 2014.
  * [54] P. G. Emma, W. R. Reohr, and M. Meterelliyoz, “Rethinking Refresh: Increasing Availability and Reducing Power in DRAM for Cache Applications,” in _MICRO_ , 2008.
  * [55] M. Ghosh and H.-H. S. Lee, “Smart Refresh: An Enhanced Memory Controller Design for Reducing Energy in Conventional and 3D Die-Stacked DRAMs,” in _MICRO_ , 2007.
  * [56] S. Kim, W. Kwak, C. Kim, D. Baek, and J. Huh, “Charge-Aware DRAM Refresh Reduction with Value Transformation,” in _HPCA_ , 2020.
  * [57] S. Liu, K. Pattabiraman, T. Moscibroda, and B. G. Zorn, “Flikker: Saving DRAM Refresh-Power Through Critical Data Partitioning,” in _ASPLOS_ , 2012.
  * [58] P. Nair, C.-C. Chou, and M. K. Qureshi, “A Case for Refresh Pausing in DRAM Memory Systems,” in _HPCA_ , 2013.
  * [59] J. Stuecheli, D. Kaseridis, H. C. Hunter, and L. K. John, “Elastic Refresh: Techniques to Mitigate Refresh Penalties in High Density Memory,” in _MICRO_ , 2010.
  * [60] T. Zhang, M. Poremba, C. Xu, G. Sun, and Y. Xie, “CREAM: A Concurrent-Refresh-Aware DRAM Memory Architecture,” in _HPCA_ , 2014.
  * [61] J. A. Mandelman, R. H. Dennard, G. B. Bronner, J. K. DeBrosse, R. Divakaruni, Y. Li, and C. J. Radens, “Challenges and Future Directions for the Scaling of Dynamic Random-Access Memory (DRAM),” in _IBM JRD_ , 2002.
  * [62] M. Redeker, B. F. Cockburn, and D. G. Elliott, “An Investigation Into Crosstalk Noise in DRAM Structures,” in _MTDT_ , 2002.
  * [63] D. S. Yaney, C.-Y. Lu, R. A. Kohler, M. J. Kelly, and J. T. Nelson, “A Meta-Stable Leakage Phenomenon in DRAM Charge Storage-Variable Hold Time,” in _IEDM_ , 1987.
  * [64] Y. Konishi, M. Kumanoya, H. Yamasaki, K. Dosaka, and T. Yoshihara, “Analysis of Coupling Noise Between Adjacent Bit Lines in Megabit DRAMs,” _JSSC_ , 1989.
  * [65] J. Liu, B. Jaiyen, Y. Kim, C. Wilkerson, and O. Mutlu, “An Experimental Study of Data Retention Behavior in Modern DRAM Devices: Implications for Retention Time Profiling Mechanisms,” in _ISCA_ , 2013.
  * [66] S. Khan, D. Lee, Y. Kim, A. R. Alameldeen, C. Wilkerson, and O. Mutlu, “The Efficacy of Error Mitigation Techniques for DRAM Retention Failures: A Comparative Experimental Study,” in _SIGMETRICS_ , 2014.
  * [67] S. Khan, C. Wilkerson, D. Lee, A. R. Alameldeen, and O. Mutlu, “A Case for Memory Content-Based Detection and Mitigation of Data-Dependent Failures in DRAM,” in _IEEE CAL_ , 2016.
  * [68] S. Khan, D. Lee, and O. Mutlu, “PARBOR: An Efficient System-Level Technique to Detect Data-Dependent Failures in DRAM,” in _DSN_ , 2016.
  * [69] S. Khan, C. Wilkerson, Z. Wang, A. R. Alameldeen, D. Lee, and O. Mutlu, “Detecting and Mitigating Data-Dependent DRAM Failures by Exploiting Current Memory Content,” in _MICRO_ , 2017.
  * [70] J. S. Kim, M. Patel, A. G. Yağlıkçı, H. Hassan, R. Azizi, L. Orosa, and O. Mutlu, “Revisiting RowHammer: An Experimental Analysis of Modern Devices and Mitigation Techniques,” in _ISCA_ , 2020.
  * [71] L. Orosa, A. G. Yağlıkçı, H. Luo, A. Olgun, J. Park, H. Hassan, M. Patel, J. S. Kim, and O. Mutlu, “A Deeper Look into RowHammer’s Sensitivities: Experiemental Analysis of Real DRAM Chips and Implications on Future Attacks and Defenses,” in _MICRO_ , 2021.
  * [72] O. Mutlu, “The RowHammer Problem and Other Issues we may Face as Memory Becomes Denser,” in _DATE_ , 2017.
  * [73] Y. Kim, R. Daly, J. Kim, C. Fallin, J. H. Lee, D. Lee, C. Wilkerson, K. Lai, and O. Mutlu, “Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors,” in _ISCA_ , 2014.
  * [74] O. Mutlu and J. Kim, “RowHammer: A Retrospective,” in _TCAD_ , 2019.
  * [75] A. G. Yağlıkçı, H. Luo, G. F. De Oliviera, A. Olgun, M. Patel, J. Park, H. Hassan, J. S. Kim, L. Orosa, and O. Mutlu, “Understanding RowHammer Under Reduced Wordline Voltage: An Experimental Study Using Real DRAM Devices,” in _DSN_ , 2022.
  * [76] O. Mutlu, A. Olgun, and A. G. Yağlıkcı, “Fundamentally Understanding and Solving Rowhammer,” in _ASP-DAC_ , 2023.
  * [77] L. Cojocar, K. Razavi, C. Giuffrida, and H. Bos, “Exploiting Correcting Codes: On The Effectiveness Of ECC Memory Against Rowhammer Attacks,” in _S &P_, 2019.
  * [78] A. G. Yağlıkçı, M. Patel, J. S. Kim, R. Azizibarzoki, A. Olgun, L. Orosa, H. Hassan, J. Park, K. Kanellopoullos, T. Shahroodi, S. Ghose, and O. Mutlu, “BlockHammer: Preventing RowHammer at Low Cost by Blacklisting Rapidly-Accessed DRAM Rows,” in _HPCA_ , 2021.
  * [79] Y. Park, W. Kwon, E. Lee, T. J. Ham, J. H. Ahn, and J. W. Lee, “Graphene: Strong yet Lightweight Row Hammer Protection,” in _MICRO_ , 2020.
  * [80] Apple Inc., “About the Security Content of Mac EFI Security Update 2015-001,” https://support.apple.com/en-us/HT204934, 2015.
  * [81] F. Brasser, L. Davi, D. Gens, C. Liebchen, and A.-R. Sadeghi, “Can’t Touch This: Practical and Generic Software-Only Defenses Against Rowhammer Attacks,” arXiv, 2016.
  * [82] R. K. Konoth, M. Oliverio, A. Tatar, D. Andriesse, H. Bos, C. Giuffrida, and K. Razavi, “ZebRAM: Comprehensive and Compatible Software Protection Against Rowhammer Attacks,” in _OSDI_ , 2018.
  * [83] V. van der Veen, M. Lindorfer, Y. Fratantonio, H. P. Pillai, G. Vigna, C. Kruegel, H. Bos, and K. Razavi, “GuardION: Practical Mitigation of DMA-Based Rowhammer Attacks on ARM,” in _DIMVA_ , 2018.
  * [84] Z. B. Aweke, S. F. Yitbarek, R. Qiao, R. Das, M. Hicks, Y. Oren, and T. Austin, “ANVIL: Software-Based Protection Against Next-Generation Rowhammer Attacks,” in _ASPLOS_ , 2016.
  * [85] E. Lee, I. Kang, S. Lee, G. Edward Suh, and J. Ho Ahn, “TWiCe: Preventing Row-Hammering by Exploiting Time Window Counters,” in _ISCA_ , 2019.
  * [86] S. M. Seyedzadeh, A. K. Jones, and R. Melhem, “Counter-Based Tree Structure for Row Hammering Mitigation in DRAM,” _CAL_ , 2017.
  * [87] M. Son, H. Park, J. Ahn, and S. Yoo, “Making DRAM Stronger Against Row Hammering,” in _DAC_ , 2017.
  * [88] J. M. You and J.-S. Yang, “MRLoc : Mitigating Row-Hammering Based on Memory Locality,” in _DAC_ , 2019.
  * [89] Z. Greenfield and T. Levy, “Throttling Support for Row-Hammer Counters,” 2016, U.S. Patent 9,251,885.