# NVMExplorer: A Framework for Cross-Stack Comparisons of Embedded Non-
Volatile Memories

Lillian Pentecost1§, Alexander Hankin2§, Marco Donato2, Mark Hempstead2, Gu-
Yeon Wei1, David Brooks1  1Harvard University, Cambridge, MA, USA
2Tufts University, Medford, MA, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The current computing landscape is dominated by data-intensive applications,
making data movement one of the most prominent performance bottlenecks. With
repeated off-chip memory access to DRAM driving up power, and SRAM technology
scaling and leakage power limiting the efficiency of embedded memories, there
is a need for new memory systems that can enable denser, more energy-efficient
future on-chip storage. The actively expanding field of emerging, embeddable
non-volatile memory (eNVM) technologies is providing many potential candidates
to satisfy this need. However, eNVM cell technologies are in vastly different
stages of development and introduce distinct trade-offs in terms of density,
read, write, and reliability characteristics.

We present NVMExplorer (http://nvmexplorer.seas.harvard.edu/): a cross-stack
design space exploration framework to compare and evaluate future on-chip
memory solutions with system constraints and application-level impacts in-the-
loop. This work uses NVMExplorer to evaluate eNVM-based storage for a range of
application and system contexts including machine learning on the edge, graph
analytics, and general purpose cache. Additionally, NVMExplorer provides an
interactive and easily navigable set of data visualizations, which allow users
to quickly answer their specific questions regarding eNVMs, filter according
to system and application constraints, and efficiently iterate and refine the
design space.

§§footnotetext: Authors contributed equally to this work.

##  I Introduction

The wide adoption of data-intensive algorithms to tackle today’s computational
problems introduces new challenges in designing efficient computing systems to
support these applications. Hardware specialization has shown potential in
supporting state-of-the-art machine learning and graph analytics algorithms
across several computing platforms; however, data movement remains a major
performance and energy bottleneck. As repeated memory accesses to off-chip
DRAM impose an overwhelming energy cost, we need to rethink the way embedded
memory systems are built in order to increase on-chip storage density and
energy efficiency beyond what is currently possible with SRAM.

In recent years, CMOS-compatible, embedded nonvolatile memory (eNVM) research
has transitioned from articles and technical reports to manufacturing flows
and product lines. These technologies hold incredible promise toward
overcoming the memory wall problem. For example, one approach inspired by
these new technologies combines the advantages of highly specialized
architectures with the benefits of non-volatile memories by leveraging analog
compute capabilities [24, 119, 129, 30]. On the other hand, the need for
optimized on-chip storage solutions and memory innovation applies both to
specialized hardware accelerators and for general-purpose CPU systems as well.
More broadly, prior works have unveiled incredible potential improvements in
storage density and energy efficiency by employing eNVMs across various
architecture domains [53, 62, 112]. With many publications showcasing the
benefits of eNVM storage technologies, it is critical for system designers to
be able to explore their varying capabilities and empower efficient future on-
chip storage. Unfortunately the architecture and broader research community
lacks a holistic tool to quantify the system and application-level
implications of memory cell technologies and to make informed decisions while
navigating the vast eNVM design space.

[capbesideposition=right,bottom,capbesidewidth=0.4]figure[]

Figure 1: Number of NVM publications from VLSI, ISSCC, and IEDM 2016-2020
(cited in text) shows strong interest in RRAM and STT and emerging
technologies, such as ferroelectric-based ones.  Figure 2: NVMExplorer
framework overview; cross-stack design space specifications and application
characteristics are evaluated in an efficient multi-stage process, then
displayed in an interactive set of data visualizations to enable informed,
application-aware comparisons of future on-chip storage solutions, as
described in more detail in Section II.

Figure 1 summarizes device and circuit conference publications relating to
eNVMs from 2016 to 2020 [70, 90, 107, 86, 68, 95, 160, 140, 14, 80, 96, 73,
31, 72, 139, 106, 147, 25, 91, 154, 115, 12, 131, 102, 58, 67, 85, 128, 123,
43, 122, 44, 84, 65, 51, 117, 151, 92, 158, 126, 134, 99, 20, 110, 55, 159,
59, 104, 3, 71, 93, 38, 75, 148, 32, 79, 137, 22, 94, 8, 143, 66, 88, 29, 161,
152, 142, 36, 47, 78, 82, 124, 157, 130, 118, 138, 150, 7, 42, 21, 132, 135,
46, 41, 48, 49, 27, 64, 57, 146, 100, 60, 77, 2, 45, 6, 105, 98, 4, 69, 56,
10, 149, 16, 136, 33, 109, 5, 87, 50, 23, 155, 97, 144, 81, 101, 39, 11, 156,
28, 125, 103, 141, 89, 26, 17, 153]. In the past five years, consistent
interest in RRAM and STT was accompanied by emerging solutions with different
physical properties such as FeFET-based memories. Each published example
offers compelling and distinct trade-offs in terms of read and write
characteristics, storage density, and reliability. In addition, the space of
eNVM technologies is constantly evolving with certain technologies moving out
of fashion or into production. Given the fluidity and complexity of this
design space, application experts and system designers need to be able to
evaluate which cell technologies are most likely to provide better efficiency,
higher storage density, or improvements on other key metrics in the context of
different computing demands. Similarly, device designers and memory architects
need high-level guidance to co-design their innovations toward more practical
and maximally beneficial future, heterogeneous memory systems.

This work introduces NVMExplorer, an end-to-end design space exploration
framework that addresses key cross-stack design questions and reveals future
opportunities across eNVM technologies under realistic system-level
constraints, while providing a flexible interface to empower further
investigations. In this work, we describe NVMExplorer and present case studies
made uniquely possible by the capabilities of NVMExplorer. In summary,
NVMExplorer makes the following key contributions:

  * •

An open-source code base including:

    * –

A database of eNVM cells described in recent literature (122 surveyed ISSCC,
IEDM, and VLSI publications) (Section III-A)

    * –

A “tentpole” methodology to summarize limits and trends across technology
classes (Section III-B)

    * –

Our end-to-end evaluation flow (Fig. 2)

    * –

Extensive source-code documentation

    * –

Many example configuration files and tutorial materials for cross-stack design
studies

    * –

An interactive web-based data visualization dashboard (Section II-C)

  * •

A unified platform to explore the viability of eNVMs in specific application
and system settings, which reveals cross-stack dependencies and optimization
opportunities, in addition to reproducing and expanding previous published
studies, (e.g., [112] [53]) (Section IV).

  * •

A unified platform to perform co-design studies of application properties,
system constraints, and devices in order to bridge the gap between architects
and device designers for future eNVM solutions. Our example co-design studies
reveal both opportunities and potential disconnects among current research
efforts (Section V).

After describing NVMExplorer (Section II), we present a snapshot of the
current eNVM landscape and extract a representative range of cell-level
behavior (Section III). Surveying recent eNVM publications reveals diverse
characteristics, highlighting the challenge in identifying solutions that
satisfy a broad range of application scenarios. Thus, Section IV presents
application-driven case studies using NVMExplorer to explore and analyze eNVM
storage solutions for DNN inference acceleration, graph processing, and
general-purpose compute. We find that each eNVM is viable in certain contexts,
and the most compelling eNVM is dependent on application behavior, system
constraints, and device-level choices. This finding suggests the existence of
many possible architecture-device co-design opportunities, which is the focus
of Section V. Finally, we differentiate NVMExplorer from related tools
(Section VI).

Figure 3: For fixed capacity (4MB) and under various optimization targets,
array-level metrics reveal each eNVM has unique, compelling attributes. Note
Pessimistic PCM write latency ($>10\mu$s) is omitted for clarity in the write
energy vs. latency plot.

##  II NVMExplorer

At a high level, NVMExplorer is a comprehensive design space exploration (DSE)
framework integrating application-level characteristics, system constraints,
and circuit and device parameters in a publicly-available, simple-to-use flow.
The overall structure of NVMExplorer (Fig. 2) relies on three stages,
described in more details in the following subsections:

  1. 1.

A comprehensive cross-stack configuration interface to specify the design
space of interest. This configuration spans the computing stack from
application (blue) and system (orange) down to circuits and devices (green).

  2. 2.

An evaluation engine which automatically generates configurations, simulates
memory arrays, processes application behavior, computes key metrics such as
performance, power, area, accuracy, and lifetime, and generates meaningful
visualizations. Evaluation steps which extend existing tools are shaded grey
in Fig. 2.

  3. 3.

An interactive, web-based visualization tool to aide discovering, filtering
and refining eNVM design points.

###  II-A Cross-Stack Configuration

To evaluate and compare eNVM solutions in system settings, it is not just cell
or even array-level characteristics of a particular technology that matter.
Rather, viable solutions depend on the area/power budget of a system and how
applications running on that system interact with the memory. NVMExplorer
provides a rich interface for configuring key application, system, and circuit
and device parameters.

At the application level, the user inputs information about memory traffic,
which may include the number of read and write operations, their proportion
relative to the total number of memory accesses, and how accesses are spread
out over execution time. These configuration parameters may be fixed values
(e.g., characterization results of a specific workload) or provided as ranges
to generate generic memory traffic patterns. Some applications may have
additional demands or metrics which are tightly related to memory technology
characteristics. For example, machine learning applications or approximate
computing methods may trade-off relaxed accuracy for performance and energy,
and NVMExplorer also provides an interface for designers to study the
application interactions and implications of fault-prone eNVM solutions.

At the system level, the user has the freedom to evaluate a wide variety of
memory configuration options by either setting performance, power, and area
constraints and optimization goals or by choosing memory array specifications
such as capacity, multi-level programming, bank configuration, and more. The
circuits and devices level of the design space configuration comprises per-
technology memory cell characteristics, in addition to sensing and programming
circuitry choices. NVMExplorer also provides a database of eNVM cell
configurations derived from ISSCC, IEDM, and VLSI publications, as described
in Section III, but it is also possible (and encouraged!) for users to extend
the current NVMExplorer database with new simulation-based (i.e. SPICE or TCAD
models), measured, or projected circuit and device properties. Once the full-
stack specifications are set, NVMExplorer automatically generates
configuration files, which are used as input to the evaluation engine.

TABLE I: High-level listing of memory cell technologies and ranges for key
characteristics; recent publications are complemented by simulation and
industry references to form technology cell definitions discussed in Section
III-A.

| SRAM | PCM | STT | SOT | RRAM | CTT | FeRAM | FeFET
---|---|---|---|---|---|---|---|---
Cell Area [$F^{2}$] | 146 | 25-40 | 14-75 | [20] | 4-53 | 1-12 |  | 4-103
Tech. Node [$nm$] | 7-16 | 28-120 | 22-90 | [1000] | 16-130 | 14-16 | 40 | 45
MLC | no | yes | yes | yes | yes | yes | yes | yes
Read Latency [$ns$] | 0.5-1.5 | [1-100] | 1.3-19 | 1.4-11 | 3.3-2e3 |  | 14 |
Write Latency [$ns$] | 0.5-1.5 | 10-3e4 | 2-200 | 0.35-17 | 5-1e5 | 6e7-2.6e9 | 14-1e3 | 0.93-1.3e3
Read Energy [$pJ$] | 1.1-2.4 |  | 0.21-1.2 |  | 1e-3 |  | 0.001 |
Write Energy [$pJ$] | 1.1-33 |  | 0.6-4.5 | [0.015-8] | 0.68 |  |  | 0.0003-0.01
Endurance [Cycles] | N/A |  105–1011 |  105–1015 |  |  103–108 |  104 |  104–1011 |  107-1011
Retention [s] | N/A |  108–1010 |  108 |  108 |  103–108 |  108 |  105–108 |

###  II-B Evaluation Engine

Given the auto-generated cell and system-level sweep configurations, the
evaluation engine produces memory array architecture characterizations and
computes application- and system-level power, performance, area, and
reliability metrics. NVMExplorer combines a customized memory array simulator,
an application-level fault injection tool, and an analytical model to
extrapolate application-level metrics.

To characterize memory arrays, we rely on a customized version of NVSim, a
previously validated tool to compute array-level timing, energy, and area
[37]. We build on existing efforts to extend NVSim to support multi-level
cells and ferroelectric-based eNVMs [120, 112]. In addition, we modified the
tool interface to ease data collection and post-processing. We introduce the
capabilities of NVMExplorer in comparing eNVMs in Section II-B1. Results of
cell-level and circuit-level simulations can be used to parameterize fault
models and perform application-level fault injection, as described in Section
II-B2. For performance estimations, in lieu of cycle-accurate simulation, we
utilize a long-pole, bandwidth driven model that takes memory access latency
and available read/write bandwidth and compares aggregated access latency per
workload execution and per second of execution to workload access statistics.
This is similar in spirit to performance models in [40, 108], and it serves
the primary purpose of identifying memory solutions that cause application
slowdown, rather than predicting precise latency implications. To extract
other critical application-level metrics, such as energy, we aggregate the
read and write access energy based on the number of application accesses and
array energy-per-access with the leakage power, scaling according to use-case
and wake-up frequency for intermittent operation. Similarly, memory lifetime
is extrapolated by comparing the average reported endurance to the write
access pattern per workload and the use-case.

####  II-B1 Example Array-Level Comparison

Figure 3 presents example array characterization output generated by
NVMExplorer after evaluating various eNVM configurations implemented in a 22nm
node. The design points are color-coded to highlight optimistic (green),
pessimistic (red), or reference (blue) designs across surveyed publications
per cell technology. The figure also reports the characteristics of 16nm SRAM
as a comparison point. For each technology, we show array characterization
under different optimization goals, which result in a variety of internal
array architectures. For example, we observe a wide range for the read-energy-
per-bit of an iso-capacity SRAM array. This result reflects the effect of
different array optimization targets (read energy-delay product, write
characteristics, area) on the internal bank configuration and periphery
overhead.

This preliminary study already provides a few key takeaways. Each eNVM is able
to attain read access characteristics competitive with SRAM, with the
exception of an array characterized with pessimistic underlying PCM cell
characteristics. However, write access characteristics vary dramatically
across published eNVM examples, in addition to the range of reported endurance
per technology. The tension between these properties and potential storage
density (even in the absence of multi-level cell programming) indicates that
array-level comparison in isolation may guide a system designer towards sub-
optimal solutions. For example, a FeFET-based memory may seem a fitting choice
for high-density, read-performant storage, but we find that both performance
and energy efficiency of those memories are highly shaped by application
traffic patterns and underlying cell assumptions. Thus, the cross-stack nature
of data exploration supported by NVMExplorer is essential in guiding system-
level choices and further investigation.

####  II-B2 Fault Modeling and Reliability Studies

In addition to characterizing memory performance, power, area, and lifetime,
NVMExplorer extends previously validated efforts in application-level fault
injection to provide an interface for fault modeling and reliability studies
[116]. Users can provide an expected error rate or more detailed, technology-
specific fault models and storage formats to perform fault injection trials on
application data stored in different eNVMs. To quantify the impact on
application-specific metrics of accuracy, the fault injection tool is tightly
integrated with application libraries for data-intensive workloads, including
PyTorch for DNNs and snap for graph processing [111, 83], as well as numpy for
generic application data. As a demonstration, we perform SPICE simulation and
extract fault charactieristics associated with single-level vs. multi-level
cell (SLC vs. MLC) programming and sensing circuitry characteristics. In this
work, we consider a subset of eNVMs, namely, RRAM, CTT, and FeFET, whose fault
characteristics could be derived from existing modeling efforts [112, 120]. We
use our extended fault injection framework to simulate the impact of storing
workload data in SLCs vs. MLCs in Section V-C. Armed with these additional
capabilities, NVMExplorer can replicate the results of previous considerations
of eNVM storage reliability [112], in addition to providing a broader platform
for studying the interactions between programming choices, cell
characteristics, and application accuracy.

###  II-C Exploring Results & Conducting Studies

The figures in this work are snapshots from NVMExplorer’s interactive web-
based data visualization tool, which will be freely available at the time of
publication of this work. In each study, we filter and constrain evaluated
results according to system optimization priorities and application use cases,
as described in the text. The basic NVMExplorer data visualization dashboard
presents power, performance, area, and memory lifetime results across all
user-configured sweep results (e.g., many application traffic patterns, array
provisioning choices, and/or eNVM cell configurations) alongside array-level
metrics for a holistic design exploration experience. A user can filter
results in terms of important constraints (e.g., latency or accuracy targets,
power or memory area budget) and identify design points of interest. While
several features of these visualizations, built using Tableau [1], are evident
in the figures in this work, including dynamic filtering across plots, click-
and-drag to narrow the design space, and pop-up details about results, we
encourage the reader to use their imagination in how they might explore and
filter the data shown in alternative ways according to their interests,
questions, or confusions.

##  III Technology Landscape

NVMExplorer provides a broad survey of published eNVM examples (Section
III-A), which can be parameterized so that systems experts can make
meaningful, high-level comparisons across technologies despite different
underlying trade-offs and maturity (Section III-B). We validate this approach
per-technology against fabricated memory arrays (Section III-C).

###  III-A Cell Definitions

We compile device- and array-level data across eNVM technologies, as
summarized in Table I alongside SRAM properties. We source the majority of the
cell-level parameters from ISSCC, IEDM, and VLSI publications and focus
primarily on works from 2017-2020 to reflect the most recent range of
achievable behavior per technology. Previous efforts detailed the physical
properties and limitations per technology [18], while NVMExplorer focuses on
compiling sufficient cell-level details and leaning on existing technology
models to provide a broad and practical database of cell definitions. While we
hope these extracted cell definitions are helpful to the community in
calibrating the current state-of-the-art, NVMExplorer is extensible as the
design space continues to evolve, as demonstrated in Section V.

The technology classes in Table I are at different levels of maturity. For
example, SOT is a relatively recent technology, and while it boasts very
impressive write speed and lower write current compared to STT, it is not yet
published at advanced process nodes. We also see that endurance varies by
multiple orders of magnitude across different technologies. Thus, adoption
will depend on the write intensity of target applications and system dynamics,
so incorporating memory lifetime estimation becomes a critical design
consideration.

Grey cells in Table I indicate parameters unavailable in recent publications.
This could be for reasons of propriety from industry fabrication or
experimental constraints. However, for architects, it is important to have
some concept of the possible range of values associated with these parameters.
In those cases where SPICE models for a technology are available, we use
simulations to fill in missing parameters. Alternatively, we consider older
publications and consult with device experts to reason about cell and array
parameters.

###  III-B Tentpoles of the Design Space

Comparing eNVMs at varying stages of development and with varying underlying
physical properties is a challenging task. The case studies in this work aim
to provide high-level guidance and relative judgments about which eNVM cell
technologies are worthy of further investigation under specific system and
application constraints. Thus, rather than focus in on specific, physically
accurate cell configurations, we aim to model the bounds of what is
conceivable per eNVM technology across the full range of published recent
academic work. We liken identifying and evaluating these bounds per-technology
to forming the poles of a tent that encompasses the full extent of eNVM
properties, so we call the extrema in terms of cell-level characteristics
(i.e., smallest, lowest read energy, best retention vs. largest cell size,
lowest endurance) the device-level “tentpoles”. In an actively evolving
technology space, this approach allows us to make meaningful classifications
about which technologies are potentially adoptable solutions. These modeling
choices are classified into two fixed cell configurations for applicable
technologies, as summarized in Section III-B1 and the figure alongside Table
I. We validate that the “tentpoles” of the cell-level design space result in
array-level characterization that provides coverage of published memory array
properties, as discussed in Section III-C.

####  III-B1 Optimistic and Pessimistic Cell Configurations

For the technology classes most represented in our survey (Fig. 1), we compute
which published example has the best-case and worst-case storage density in
terms of Mb/F2, and this data serves as the foundation of the bounds of the
cell-level design space; those points which are most and least dense across
recent published examples. Any critical cell-level parameters not reported
with those cell definitions are assigned values (e.g., read characteristics
and programming settings) using the best (lowest power, highest efficiency) or
worst (highest power, lowest efficiency) value per metric across all other
recent publications with sufficient supporting data. These best-case and
worst-case technologies per class form the tentpoles of the underlying cell
design space, and we label these fixed cell definitions as “optimistic” or
“pessimistic” accordingly. For the purposes of the case studies presented in
Sections IV and V, all array- and application-level results are produced using
these fixed underlying optimistic and pessimistic cell properties, though we
note that a user of NVMExplorer can draw either on these constructed, bounding
example cells or on the full database of surveyed configurations, or on fully
customized definitions with respect to cell size, access properties, and
operating conditions (e.g.,read/write voltage, temperature). Corresponding
fault models and error rates for reliability studies are extracted after
optimistic vs. pessimistic cell-level properties are fixed, as discussed in
one of the presented case studies (Section V-C).

This approach helpful for many reasons: for one, these extremes help us answer
exploratory questions about what we will likely see in the near future;
secondly, comparing the best-case of one technology to the worst-case of
another can help gauge less mature technologies against more mature reference
points; thirdly, if such optimistic configurations are untenable or even
pessimistic configurations are attractive in a specific system setting, we can
build confidence for further exploration and more detailed modeling efforts
without implementing and attempting to meaningfully compare many many cell
definitions with insufficient data. A limitation of this methodology is that
inherent trade-offs between certain parameters for a technology may not be
linked (e.g., area, latency, and retention for STT); however, this amalgam of
cell properties represent the full spectrum of achievable characteristics per
technology, rather than specific fabricated results. As a point of additional
comparison, the results shown in the following studies include a reference
cell configuration for RRAM as a relatively mature eNVM, with parameters
derived from a specific industry result [29]. The resulting optimistic,
pessimistic, and reference cell size and write pulse are shown to the right of
Table I.

Figure 4: “Tentpole” STT vs. published array data shows coverage of the space
across critical metrics.

###  III-C Validation

Our array-level area, energy, and latency characterizations rely on the
previously-validated procedures of NVSim to extrapolate cell-level
configurations and array design constraints to optimized memory layouts and
properties [37]. However, in employing our “tentpole” approach, it is critical
that we verify that array-level results using our optimistic and pessimistic
underlying cell characteristics fully cover and match expectations of existing
fabricated eNVM solutions.

Whenever possible, we select publications with array-level characterizations
for a given technology, and compare those results to iso-capacity memory
arrays modeled through our “tentpole” approach. Figure 4 shows an example of
such an exercise. We compare a 1MB STT-RAM array published at ISSCC in 2018 to
optimistic and pessimistic STT design points produced by NVMExplorer. Here, we
note that our tentpole results effectively represent the range of actual array
properties by producing metrics that are both higher and lower, but similar in
magnitude, to the reference STT-RAM array. The studies presented in this work
consider only validated configurations for which we were able to either
complete this validation exercise or run SPICE-level simulations. It is worth
noting that NVMExplorer is set up to evaluate all cell technologies in Table I
(e.g., though SOT is a compelling emerging solution and NVMExplorer users can
configure and evaluate SOT-RAM, our survey found insufficient array-level data
for validation, so it is omitted in Section IV and V). System validation and
application characteristics are derived from existing, state-of-the-art
references, as addressed in each study in Section IV.

##  IV Application-Driven Case Studies

We now present three case studies that highlight different ways NVMExplorer
can search design spaces in order to identify benefits and limitations of the
diverse range of eNVM storage solutions. Each scenario presents unique
optimization goals and system priorities and, in each case, we compare how
each eNVM’s power, performance, and area fairs relative to similarly-
provisioned SRAM or DRAM in a baseline system.

Figure 5: Read characteristics and storage density for 2MB arrays, provisioned
to replace on-chip SRAM for NVDLA.

Figure 6: The most energy-efficient eNVM varies under different DNN inference
use cases, such as continuous (left, operating power) vs. intermittent (right,
reporting energy per input image frame); these results exclude eNVM solutions
that are unable to meet application latency and accuracy targets.

###  IV-A DNN Inference Accelerator

Prior studies have demonstrated the potential benefits of eNVM storage for
Deep Neural Network (DNN) inference accelerators [112, 34, 145], albeit with
limited scope in terms of eNVM technologies and cross-stack parameters
considered. NVMExplorer empowers researchers to approach a broader set of
questions that compare eNVMs in different storage scenarios (e.g., limited to
weights vs. storage of DNN parameters and intermediate results) and system
constraints (e.g., strict area budget, or power budget). In this work, we
consider two distinct use cases for a DNN inference accelerator: continuous
operation, as in image processing per frame of a streamed video input, and
intermittent operation, where the system is woken up per inference task and
can leverage the non-volatility of eNVM by retaining DNN parameters on-chip in
power-off state between inferences.

Figure 7: The eNVM storage solution (iso-capacity arrays provisioned per task,
optimized for ReadEDP) that minimizes total memory energy consumption varies
according to system wake-up frequency and DNN inference task; All solutions
shown maintain application accuracy and a $<1s$ latency per inference. Figure
8: Memory power, latency, and projected lifetime for generic traffic patterns
encompassing graph processing demands, including specific graph kernels as
labeled. The lowest power solution depends on the expected read traffic. FeFET
solutions fail to match SRAM performance. STT provides superior performance
and memory lifetime.

####  IV-A1 Continuous Operation

We consider the commonly-used and well-studied NVDLA [127] as a base computing
platform and compare its 2MB SRAM with iso-capacity eNVMs. We use the NVDLA
performance model [108] to extract realistic memory access patterns and
bandwidth requirements of the on-chip buffer. More specifically, we evaluate
the power and performance of accesses to on-chip memory storing ResNet26
weights for single-task image classification using the ImageNet dataset vs.
multi-task image processing, comprising object detection, tracking, and
classification, at a consistent frame rate of 60 frames-per-second, as is
typical for HD video. We additionally consider the impact of storing
activations in eNVM, but this ostensibly ignores endurance limitations.

First, we observe the read and storage density characteristics for 2MB arrays
using the cell-level tentpoles of several promising eNVM technology classes,
as shown in Figure 5 compared with SRAM. Notice that read energy effectively
divides arrays into two tiers. STT, PCM, and RRAM offer lower read energies
and competitive read latencies vs, SRAM. In contrast, FeFET-based eNVMs suffer
from higher read energies, but optimistic FeFET offers the highest storage
density with low latency. At similar low latency, optimistic STT offers
6$\times$ higher density over SRAM. PCM and RRAM outperform SRAM in terms of
both read latency and storage density. While such comparative insights can
readily be extracted from this pair of plots, there are other important
dimensions to also consider, and NVMExplorer facilitates more comprehensive
analyses that consider the impact of application priorities and system-level
use cases on eNVM design decisions.

Figure 6 (left) summarizes total operating power (both dynamic access and
leakage power) for the 2MB memory arrays characterized in Figure 5 and
accessed according to traffic patterns of different ResNet deployment
scenarios, i.e., single- vs. multi-task and weights-only vs. storing both
weights and activations. These results exclude eNVM candidates that cannot
support 60 FPS operation nor maintain DNN accuracy targets. Recall NVMExplorer
includes fault injection wherein high eNVM fault rates can degrade model
accuracy to unacceptable levels. While not explicitly shown here, NVMExplorer
exposes numerous additional interactions for users to probe and build
intuition. For example, while total memory power increases as the number of
accesses per frame increases to compute multiple tasks, the ratio of read-to-
write traffic stays roughly the same. Hence, the relative power of eNVM arrays
also remains similar. In particular, PCM, RRAM, and STT all offer over
4$\times$ reduction in total memory power over SRAM. One important reason for
this is that SRAM leakage power will dominate compared to eNVM solutions, even
under high traffic. Of the energy-efficient solutions, STT offers best
performance (lowest application latency per frame). In contrast, optimistic
FeFET offers higher storage density while maintaining 60FPS and a
1.5-3$\times$ power advantage over SRAM.

TABLE II: Summary of preferred eNVM under varying DNN use case, task, storage
strategy, and optimization priority.

Use Case | Inference Task | Data Storage | Priority | Opt. eNVM | Alt. eNVM
---|---|---|---|---|---
Continuous (60IPS) |  Single-Task Image Classification | Weights Only | Low Power | PCM | PCM
High Density | FeFET | CTT
Weights + Acts | Low Power | PCM | RRAM
High Density | STT | RRAM
Multi-Task Image Processing | Weights Only | Low Power | PCM | RRAM
High Density | FeFET | CTT
Weights + Acts | Low Power | STT | RRAM
High Density | STT | RRAM
Intermittent (1IPS) |  Single-Task Image Classification | Weights Only | Low Energy/Inf | RRAM | RRAM
High Density | FeFET | CTT
Multi-Task Image Processing | Weights Only | Low Energy/Inf | FeFET | FeFET
High Density | FeFET | CTT
Sentence Classification (ALBERT) | Embeddings Only | Low Energy/Inf | RRAM | RRAM
High Density | FeFET | CTT
All Weights | Low Energy/Inf | STT | RRAM
High Density | FeFET | CTT
Multi-Task NLP (ALBERT) | All Weights | Low Energy/Inf | STT | RRAM
High Density | FeFET | CTT

####  IV-A2 Intermittent Operation

Let us now consider eNVM storage for two additional use cases that alter
system-level optimization goals and corresponding eNVM selection, further
highlighting the flexibility and ease of exploration the NVMExplorer framework
offers. A major advantage of storing DNN weights in eNVMs is that non-
volatility supports intermittent operation that powers off the accelerator
between inferences. Using SRAMs would either consume leakage power to keep the
weights memory powered on or consume power to restore the weights from off-
chip memory, e.g., by incurring a latency and energy penalty by fetching from
DRAM. In this use case, we provision monolithic eNVM storage to hold all DNN
weights (e.g., up to 32MB for Natural Language Processing (NLP) tasks). For
image processing, all weight memory accesses are to eNVM, eliminating the
wake-up latency and power associated with loading parameters on-chip, in
addition to reducing distance between compute system and higher-capacity
memory.

Figure 6 (right) compares the resulting memory-energy-per-inference across
eNVMs for both single-task image classification and multi-task image
processing, as determined by the total number of accesses to retrieve all DNN
weights over the course of processing one input frame. The lowest-energy
technology choice differs between the single vs. multi-task inference and,
perhaps more interesting, both are eNVM candidates with lower storage density
(RRAM and pessimistic FeFET), as opposed to the highest density options (STT
and optimistic FeFET), which hints at a cross-stack prioritization of read
performance as opposed to cell size reduction, as probed further in Sec. V-B.
We repeat this study for single task vs. multi-task natural language
processing using the ALBERT network, a relatively small-footprint, high-
accuracy, transformer-based DNN [76].

To further study this result, we dig into the implications of intermittent
operation and compare the total energy versus the number of inferences per
day, showing a continuum of wake-up frequency that may arise (e.g., deployed
solar-powered agricultural sensors or satellites, or a voice-enabled assistant
executing NLP tasks on wake-up). The left plot of Figure 7 shows total memory
energy as a function of inferences per day for image classification. Here,
total memory energy is presented as a proxy for device battery life. From the
figure, we observe that when the number of inferences per day is sufficiently
low (less than 1e5), optimistic FeFET yields the lowest energy. At higher
wake-up frequency, optimistic STTs take over because of the relatively lower
energy-per-access. Figure 7 (right) investigates the impact on an NLP
workload. While results are similar, optimistic STT emerges as the best
technology at lower inference rates (as compared to image classification),
because ALBERT requires more computational power per inference than ResNet26.

Table II summarizes the preferred eNVM technology across different use cases
and tasks, with “Opt. eNVM” indicating the preferred choice under optimistic
underlying cell characteristics and “Alt. eNVM” indicating the preferred
technology assuming pessimistic assumptions and reference points, and table
entries for intermittent operation are selected at a fixed wake-up rate.
Across a range of device wake-up frequencies and per-wake-up compute patterns,
we observe that several eNVMs become compelling, and the preferred NVM choice
for further investigation varies depending on both of these factors.

Figure 9: Memory operating power, latency, and projected lifetime under
continuous operation across SPEC benchmark traffic to a 16MB LLC shows
preferred eNVM depends on traffic demands and optimization goal. All solutions
shown meet per-benchmark read/write demands. For high-traffic benchmarks, STT
provides lowest power, lowest latency, and longest projected lifetime.

###  IV-B Enabling Efficient Graph Processing

Our second case study explores the potential benefits of using eNVMs for graph
processing, which imposes an entirely different set of constraints in terms of
memory read and write characteristics. Graph processing comprises many read-
dominated tasks with less predictable data reuse than DNNs (e.g., search
kernels), but still involves write traffic and, overall, is incredibly data-
intensive in terms of memory bandwidth and capacity. As an initial exploration
of compatibility and viability between graph processing workloads and eNVM
storage solutions, we consider the total power and resulting memory lifetime
per technology under generic traffic patterns covering the range of read and
write bandwidths for critical graph tasks, as described in previous workload
characterization efforts [9]. As a proof of concept in a specific system, we
additionally evaluate eNVM storage solutions under access patterns for
benchmarks executed on a domain-specific accelerator [52].

####  IV-B1 Analysis for generic traffic patterns

We consider different memories experiencing a range of generic traffic
patterns representing graph processing kernels (i.e., read access rates from
1-10GB/s and write access rates from 1-100MB/s) [9]. NVMExplorer provides a
wide array of critical metrics to compare and user-configurable visualizations
to extract important trends and limitations. For example, in Figure 8, we
choose to display total memory power against read traffic, as number of read
accesses becomes a dominant factor in total power for read-dominated
workloads, and total memory latency against write traffic, as overall
performance for several eNVMs is strongly determined by write traffic.

As shown by Figure 8, left, total memory power generally increases with read
access rate and the lowest power solution depends on the application traffic
load. For applications that exhibit fewer than 107 read accesses per second,
optimistic FeFET is a clear winner, while pessimistic FeFET and RRAM are next
best candidates. On the other hand, for higher rates of read traffic (e.g.,
$>10^{8}$), optimistic STT is best. For mid-range read access rates, PCM and
RRAM are also viable solutions sometimes offering the lowest power solution.
However, this relationship alone does not dictate memory technology choice. A
slightly different and more consistent story emerges when we analyze the
impact of different eNVMs on overall memory latency (both read and write)
versus write access rates, shown by the middle plot of Figure 8. While there
is a clear preference for optimistic STT, RRAM and optimistic PCM are also
worth considering. In contrast, most pessimistic eNVM technologies and all
FeFET-based solutions are significantly inferior, even failing to match SRAM
performance for many traffic patterns.

When we additionally consider projected memory lifetime, STT emerges the clear
winner overall. Note that the right chart of Figure 8 plots the memory
lifetime assuming continuous operation at a particular write access rate.
Hence, the highest write traffic always yields the lowest lifetime. While RRAM
seemed promising based on performance and power, it has the worst endurance
and lowest lifetimes.

####  IV-B2 Analysis for domain-specific systems

In addition to relying on generic traffic to represent the full range of
expected load of graph processing, NVMExplorer can also be leveraged to answer
a more specific design question: For performance targets and traffic patterns
to a specific storage resource in a graph processing accelerator system, which
eNVMs offer compelling characteristics that warrant further investigation? To
this end, Figure 8 also includes points, identified in pink, corresponding to
memory traffic to run breadth-first search on two different social network
graphs [83]. Traffic patterns are extracted from throughput and accesses
reported for the compute stream of a domain-specific graph processing
accelerator utilizing an 8MB eDRAM scratchpad [52]. In the baseline system,
about $90\%$ of the energy is spent on the eDRAM scratchpad (not including
DRAM controller energy), with an operating power of at least 3.1W at the 32nm
process technology node as reported from Cacti [52, 54]. We analyze the
benefits of replacing the 8MB eDRAM scratchpad with an iso-capacity eNVM array
provisioned to meet the cited latency target (1.5ns).

If we exclude RRAM due to low lifetime projections, FeFET, PCM, and STT all
offer significantly lower memory power (about 2-10$\times$ lower than SRAM)
and even pessimistic STT offers consistent performance. These observations,
based on a realistic graph processing use case extracted from prior work, are
consistent with the results generated using generic traffic patterns. Again,
optimal technology choice depends on higher, system-level optimization goals,
and NVMExplorer provides critical insights in the presence or absence of a
specific system solution and simulation results.

If the high-level goal is to maximize storage density, FeFET is highly
attractive, but severely limited by poor write latency (unable to meet
application latency expectations under the higher range of traffic patterns).
Rather than prematurely eliminating FeFET, designers can leverage NVMExplorer
to study the impact of relaxing or adapting application targets or to explore
co-design solutions that target improvements to the underlying technology
(Sec. V-A) or architecture (Sec. V-D).

###  IV-C Non-Volatile LLC Solutions

Improved density and energy efficiency could revolutionize general-purpose on-
chip storage, and recent efforts have endeavored to replace high-performance
memories, like caches, with eNVM-based alternatives [74, 53, 62]. However,
caches must handle a large volume of writes depending on the application, so
the achievable write latency and endurance per eNVM comes to the forefront of
design considerations.

Figure 10: Array access characteristics in isolation for consideration of
replacing (iso-capacity) a 16MB LLC.

In this study, we consider the last-level cache (LLC) of a high-performance
desktop processor, similar to Intel’s 14nm, 8-core Skylake. The memory
hierarchy includes a private 32 KiB L1I$; a private 32 KiB L1D$; a private 512
KiB L2$ (non-inclusive, write-back); and a shared ring 16MiB L3$ with 64 B
line, 16 ways (inclusive and write-back). The system includes DRAM with 2
channels, 8 B/cycle/channel, 42cycles + 51 ns latency. Representative
application behavior comes from SPECrate CPU2017 (integer and floating point),
and we warm-up the cache for 500M instructions and simulate for 1-billion
instructions in detail using the Sniper simulator [13, 15]. This provides
application modeling data for a 16MB LLC (e.g., reads, writes, execution time
per benchmark) that are inputs to NVMExplorer (see Section II-A).

First we focus on the array characteristics of the different memory
technologies in isolation, as shown in Figure 10. From the left plot, we note
a competetive range of read energy and read latency does not reveal a clear
winner. For example, if read energy per access is highest priority, FeFET,
RRAM or even SRAM offer array configurations that trade access latency for
energy efficient, while STT and optimistic FeFET offer pareto-optimal read
characteristics. For writes (Figure 10, right), a PCM-based last level cache
appears to minimize energy per access. On the other hand, only STT and RRAM
are able to beat SRAM write latency. Again, we find array characteristics in
isolation do not offer sufficient guidance to choose the best eNVM for LLC,
and NVMExplorer allows us to go further.

Figure 9 shows the resulting power, performance, and lifetime when using
different eNVMs as LLC and assuming memory traffic from SPEC2017 benchmarks.
The leftmost figure shows total memory power versus read access rate, where
each column of points corresponds to a particular benchmark traffic pattern.
We again see that the lowest power eNVM solution depends on the traffic
pattern. In broad terms, RRAM and FeFET fair better for lower read access
rates while PCM is better for higher rates until STT emerges best for the
highest rates. In terms of memory access latency with respect to write access
rates, STT is usually the best choice, though arrays unable to meet
application bandwidth are excluded. Lastly, the rightmost figure compares
lifetimes across the eNVM technologies for a range of write access rates.
Again, STT offers the best longevity on average. However, PCM and FeFET may
warrant consideration for read-dominated workloads. RRAM, on the other hand,
does not appear viable as an LLC.

##  V Co-Design Opportunities

Exploration of the design space in Section IV shows that no single eNVM
technology is best. Rather, technology choice depends on the application and
system-level targets. This also means there are ample co-design opportunities
across the computing stack – from devices to architecture.

Figure 11: Back-gated (BG) FeFETs provide the high density and low operating
power for example graph processing benchmarks with SRAM-comparable performance
and begin to close the performance gap between non-BG FeFET and other memory
technologies across SPEC2017 benchmarks.

###  V-A Alternative FeFET fabrication choices unlock performant solutions for
graph processing

Previous FeFET-based device characterization and modeling efforts have
exhibited write pulses on the order of $100ns$-$1\mu s$. However, alternative
FeFET fabrication strategies in early development stages, such as back-gated
FeFETs [121], offer compelling potential advancements in write latency ($10ns$
programming pulse) and projected endurance ($10^{12}$). Section IV-B noted
that the primary limition of FeFETs in the context of graph processing was an
inability to meet the application latency targets under higher write traffic.
Thus, using the underlying cell properties of back-gated FeFETs reported in
[121], we can rapidly re-examine the viability of FeFET-based memory and probe
whether this change could make a difference in the viability of FeFET-based
memory for graph processing.

Figure 11 shows the total memory power and total memory latency of an 8MB
memory array of back-gated FeFETs (in yellow) compared to using previous FeFET
standards (red, green) and SRAM (blue). We examine these metrics under a range
of read and write traffic patterns which are inclusive of the graph benchmarks
described in Section IV-B and the SPEC benchmarks used in Section IV-C, but
here showing access patterns for an 8MB capacity LLC. The underlying array-
level characterization is shown in Figure 11, right. From the array
characterization, we observe that the back-gated FeFETs show a slight increase
in read energy per access and slight decrease in storage density compared to
prior state-of-the-art cells. However, we observe that they enable comparable
application latency to SRAM across a wide range of write traffic where
previous FeFET versions fall short. Furthermore, back-gated FeFETs results in
the lowest operating power over most of the range of read accesses per second,
including for the example graph processing benchmark, Wikipedia–BFS8MB.

Based on these observations, we posit that back-gated FeFET memory may close
the performance gap between prior FeFETs and other memory technologies
(including SRAM) and unlock additional application domains. NVMExplorer’s
ability both to quickly and efficiently gauge the impact of cell-level
innovations and to match emerging device designs to compelling use cases can
enable productive future co-design collaborations. This feedback loop is
mutually beneficial in providing direct motivation for further device
development and encouraging system designers to integrate more energy-
efficient, highly dense on-chip memory.

Figure 12: Results for 8MB arrays are filtered according to a maximum area
efficiency (top right). Arrays with lower area efficiency are highlighted
across all views and tend to result in low memory latency across many traffic
scenarios.

###  V-B Trade area efficiency for performance

One theme we can highlight across the architecture-driven case studies from
Section IV is that the subset of characterized results that exhibit lower area
efficiency (i.e., internal array architectures that do less amortization of
periphery and sensing overhead) also tend to result in lower total memory
latency across many traffic scenarios. This is perhaps counter-intuitive given
the effort spent in the devices community to manufacture very small cell
sizes. We also note that in Figure 12, where such design points are
highlighted across the plots, that slight advantages in terms of energy-per-
access (e.g., Opt. STT and PCM compared to FeFET) tend to correlate to large
total power advantages in high-traffic scenarios. As such, pointing out to
device designers the greater relative impact of reduced energy per access
rather than decreased cell size could usher in a more productive, product-
ready set of eNVM technologies. Additionally, we observe that reducing energy
per write access for STT and RRAM would drastically improve their relative
power advantage for data-intensive applications, even at a cost of relatively
lower area efficiency or storage density.

###  V-C Multi-Level Cell (MLC) advantages vary among eNVMs

While programming multiple bits per memory cell is an important strategy for
increasing storage density across many eNVMs, previous work has revealed that
MLC eNVMs may exhibit significantly higher fault rates that must be carefully
considered in conjunction with application resilience [112]. NVMExplorer
enables efficient and broad probing of reliability vs. storage density by
providing an application-agnostic fault injection tool and templates for
technology-specific fault modes (Section II-B2). To demonstrate, we quantify
the application accuracy for ResNet18 image classification under weight
storage in SLC vs. 2-bit MLC across multiple technologies for which there
exists sufficient cell and circuit level data to produce detailed fault
models. The density vs. reliability trade-off is distinct for each technology.
For example, Figure 13 displays 8MB and 16MB characterized arrays, including
2-bit MLC RRAM and 2-bit MLC FeFET, filtered such that only those arrays
meeting application latency requirements and maintaining image classification
accuracy are included. Note that these results replicate previous efforts that
indicate that image classification inference is robust to 2-bit MLC RRAM
storage (we also verified this for CTT-based memories with fault modeling
details provided in [112, 35]), while we show that MLC FeFET devices only
exhibit acceptable accuracy for larger cell sizes. This is because smaller
FeFETs are more difficult to program reliably due to device-to-device
variation [120].

Figure 13: When we consider multi-level cells (MLC)s and filter out memory
solutions that don’t provide acceptable ResNet18 inference accuracy after
fault injection, we note MLC RRAM is denser and more performant than SLC RRAM,
while MLC FeFET is only sufficiently reliable for larger cell sizes (red).

###  V-D Write buffering changes the performance landscape

In conjunction with technology innovations to reduce write latency, adoption
of a wider set of eNVMs in general-purpose computing contexts could be made
possible by employing existing architectural techniques to mask poor write
characteristics. For example, in an effort to extend memory lifetime and mask
the performance impact of write access, a more performant technology (e.g.,
SRAM, or STT) could be employed as a write-buffer. Rather than employ a costly
and engineering-intensive cycle-accurate simulator to gauge the impact of
provisioning a write buffer, NVMExplorer enables an analytical study under
user-specified traffic patterns to narrow the space of eNVMs worthy of further
simulation and design effort. This approach answers high-level questions
regarding whether write-buffering could make a difference in making additional
eNVMs viable for applications with significant write traffic, and, if so, how
much benefit would need to be extracted using the write buffer?

Figure 14: Masking write latency or reducing write traffic via introduction of
a write caching scheme could enable a broader set of eNVM technologies. TABLE
III: NVMExplorer leverages existing efforts by extending NVSim, while enabling
cross-stack DSE across multiple use cases and domains, including more breadth
than previous works, and providing a unified platform to explore and iterate
design.

|  | Tech. Surveys | Array Simulators | Arch-Specific Frameworks | This Work
---|---|---|---|---|---
|  |  IRDS [61] |  Mem. Trends [133] |  NVSim[37] |  DESTINY[114] |  Neuro- Sim+[19] |  NVMain[113] |  Deep- NVM++[62] |  NVMExplorer
|  RRAM | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | ✓
|  STT |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  SOT |  | ✓ |  |  |  |  | ✓ | ✓
|  PCM |  | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓
|  CTT |  |  |  |  |  |  |  | ✓
|  FeRAM | ✓ | ✓ |  |  |  |  |  | ✓
NVM |  FeFET |  | ✓ |  |  | ✓ |  |  | ✓
|  MLC |  |  |  |  | ✓ |  |  | ✓
Circuits |  Fault Modeling |  |  |  |  | ✓ |  |  | ✓
Architectural Simulator / Use Case |  |  |  |  |  Focus on PIM for DNNs |  gem5 |  GPGPU- sim for DNNs |  Analytical; CPU, GPU, accelerator included
App-Aware Evaluation |  Accuracy |  |  |  |  | ✓ |  |  | ✓
Memory Lifetime |  |  |  |  |  | ✓ |  | ✓
Operating Power |  |  |  |  | ✓ | ✓ | ✓ | ✓
Latency |  |  |  |  | ✓ | ✓ | ✓ | ✓

For illustrative purposes, we consider a simple write cache that would hold
write requests to the eNVM, write back to eNVM when the buffer is full, and
allow in-place updates in the case of multiple writes to the same address
before an update to eNVM. Figure 14 shows the results for this study for
SPEC2017 and Facebook-Graph-BFS. Just buffering the writes will mask the
effective write latency experienced by the system, while a write cache that
allows updates could additionally reduce traffic and extend lifetime. In
particular, we look at the effects of masking write latency and reducing write
traffic on total memory latency and power. We observe that for Facebook-Graph-
BFS, if the write traffic load is reduced by at least half, FeFET emerges as a
performant option, while STT remains the lowest power solution for this
particularly high-traffic workload. STT and RRAM are still the optimal
technology choices for SPEC2017 in terms of performance, but write-buffering
could empower FeFETs as a lower-power alternative if latency could be masked
or write traffic to the eNVM could be reduced by at least 25%.

##  VI Related Work

Previous work in evaluating eNVMs can be characterized as either focusing on
device- and array-level evaluations, or providing in-depth cross-stack
analysis for a particular combination of eNVM and application target. In Table
III, we codify the key differences between NVMExplorer and related works.
Survey works such as the Stanford Memory Trends [133] maintain a list of key
parameters, like storage capacity and write energy, while previously validated
array-level characterization tools, such as NVSim [37], characterize timing,
energy, and area of eNVM-based memory structures. DESTINY [114] modifies NVSim
to evaluate 3D integration and could be similarly extended and used as a back-
end characterization tool for NVMExplorer.

To evaluate eNVMs in a system setting, prior work typically integrates NVSim
with a system simulator. DeepNVM/DeepNVM++ [63, 62] enables cross-layer
modeling, optimization, and design space exploration of MRAM-based
technologies in the context of GPU cache for DNNs using GPGPUSim. NVMain [113]
enables evaluation of eNVM-based main memory using gem5. NeuroSim+ [19]
focuses on evaluation of processing-in-memory for DNN inference and training.
While these frameworks are great examples of domain-specific explorations and
evaluations, NVMExplorer can evaluate a variety of system and application
domains, in addition to offering reliability analysis, additional metrics such
as memory lifetime, and a database of technology cell characteristics and
configurable device parameters.

In contrast, NVMExplorer offers more breadth by including application-,
system-, and device-level considerations, and accommodating a wider range of
devices without requiring a separate system simulator. Additionally,
NVMExplorer offers a broad range of evaluations, including fault modeling and
reliability studies. It is built for ease of navigation and fluidity, and it
exposes the unique cross-stack trade-offs among application characteristics,
system constraints, and circuit and device level innovations in a user-
friendly configuration interface and companion data visualization interface.
By integrating these components, NVMExplorer additionally provides a platform
for architects and device designers to perform co-design evaluations required
for the advancement of technologically-heterogeneous memory systems.

##  VII Conclusion

Next-generation on-chip memory will need to push the boundaries of efficiency
and density, and a diverse set of embedded non-volatile memory (eNVM)
technologies have compelling characteristics to address these limitations.
NVMExplorer provides architects the flexibility to explore and compare these
storage solutions under realistic constraints. NVMExplorer is open source,
with interactive data visualizations freely available online, which we hope
will unlock the potential of eNVMs in a broad range of systems.

##  Appendix A Artifact Appendix

###  A-A Abstract

NVMExplorer is an open-source framework for modeling, evaluating, and
comparing embedded non-volatile memory solutions under different application
and system-level properties and constraints. NVMExplorer’s code base includes
(1) a python-based user interface for configuring and running design sweeps,
(2) a modified and extended version of NVSim for memory array
characterization, (3) a python-based application-level fault injection tool
with a stand-alone interface, (4) scripts to both generate and parse
associated configuration and output files from memory characterization, and
(5) an analytical model extrapolates array-level data according to user-input
application and system properties and constraints. This release also includes
(1) a per-technology database of properties extracted from paper survey of
IEDM, VLSI, and ISSCC 2016-2020, (2) application characteristics for the
workloads in our submission, including graph search, neural networks, and SPEC
CPU2017, (3) fault model characteristics and data format transformations for
fault injection studies, and (4) sample configuration files and customized
cell-level characteristics.

NVMExplorer was developed and validated on both Mac and Ubuntu Linux systems,
with successful configuration and some tests also on Windows. Support for more
advanced technology nodes and alternative memory characterization backends is
under development.

###  A-B Artifact check-list (meta-information)

  * •

Compilation:

        $ cd nvmexplorer_src/nvsim_src
    $ make

  * •

Execution:

        $ python run.py config/main_dnn_study.json

  * •

Output:

        output/results/[eNVM]_1BPC-combined.csv

  * •

How much disk space required (approximately)?: 37MB

  * •

Preparation Time?: Less than 1 hour.

  * •

Execution Time (for this artifact)?: about 1 hour (desktop CPU).

  * •

Publicly available?: Yes

  * •

Code licenses (if publicly available)?: MIT

  * •

Archived (provide DOI)?: https://zenodo.org/badge/latestdoi/375786583

###  A-C Description

####  A-C1 How to access

[Zenodo] for current release, [Github] for up-to-date and development
versions.

####  A-C2 Hardware dependencies

Tested on a selection of laptop and desktop setups; no specific HW
dependencies required.

####  A-C3 Software dependencies

Python 3.8; pandas, numpy, (optional) pytorch for fault injection experiments;
gcc

####  A-C4 Data sets

Please see provided workload characterization results and default
configuration settings in the following paths from the main NVMExplorer
repository:

     config/README.md

     data/workload_data

     output/NVM_data

###  A-D Installation

    $ git clone --recurse-submodules
    https://github.com/lpentecost/NVMExplorer
    $ cd nvmexplorer_src/nvsim_src
    $ make

Prior to running NVMExplorer, please verify you are using Python 3.8 and have
the pandas and numpy packages available.

###  A-E Experiment workflow

The general usage of NVMExplorer is via passing a JSON config file that
specifies your desired design sweep to run.py:

    python run.py config/[config name].json

For example, to generate the DNN-focused case study in Section 4.1, you can
run:

    python run.py config/main_dnn_study.json

and verify that the per-eNVM-technology CSV outputs generated are consistent
with those provided in

    AE_dnn_output/[eNVM]_1BPC-combined.csv

## References

  * [1] “Business intelligence and analytics software.” [Online]. Available: https://www.tableau.com/
  * [2] S. Aggarwal, H. Almasi, M. DeHerrera, B. Hughes, S. Ikegawa, J. Janesky, H. K. Lee, H. Lu, F. B. Mancoff, K. Nagel, G. Shimon, J. J. Sun, T. Andre, and S. M. Alam, “Demonstration of a reliable 1 gb standalone spin-transfer torque mram for industrial applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [3] M. Alayan, E. Vianello, G. Navarro, C. Carabasse, S. L. Barbera, A. Verdy, N. Castellani, A. Levisse, G. Molas, L. Grenouillet, T. Magis, F. Aussenac, M. Bernard, B. DeSalvo, J. M. Portal, and E. Nowak, “In-depth investigation of programming and reading operations in rram cells integrated with ovonic threshold switching (ots) selectors,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [4] T. Ali, P. Polakowski, K. Kuhnel, M. Czernohorsky, T. Kampfe, M. Rudolph, B. Patzold, D. Lehninger, F. Muller, R. Olivo, M. Lederer, R. Hoffmann, P. Steinke, K. Zimmermann, U. Muhle, K. Seidel, and J. Muller, “A multilevel fefet memory device based on laminated hso and hzo ferroelectric layers for high-density storage,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [5] T. Ali, K. Seidel, K. Kuhnel, M. Rudolph, M. Czernohorsky, K. Mertens, R. Hoffmann, K. Zimmermann, U. Muhle, J. Muller, J. Van Houdt, and L. M. Eng, “A novel dual ferroelectric layer based mfmfis fefet with optimal stack tuning toward low power and high-speed nvm for neuromorphic applications,” in _IEEE Symposium on VLSI Technology_ , 2020\.
  * [6] J. G. Alzate, U. Arslan, P. Bai, J. Brockman, Y. J. Chen, N. Das, K. Fischer, T. Ghani, P. Heil, P. Hentges, R. Jahan, A. Littlejohn, M. Mainuddin, D. Ouellette, J. Pellegren, T. Pramanik, C. Puls, P. Quintero, T. Rahman, M. Sekhar, B. Sell, M. Seth, A. J. Smith, A. K. Smith, L. Wei, C. Wiegand, O. Golonzka, and F. Hamzaoglu, “2 mb array-level demonstration of stt-mram process and performance towards l4 cache applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [7] F. Arnaud, P. Zuliani, J. P. Reynard, A. Gandolfo, F. Disegni, P. Mattavelli, E. Gomiero, G. Samanni, C. Jahan, R. Berthelon, O. Weber, E. Richard, V. Barral, A. Villaret, S. Kohler, J. C. Grenier, R. Ranica, C. Gallon, A. Souhaite, D. Ristoiu, L. Favennec, V. Caubet, S. Delmedico, N. Cherault, R. Beneyton, S. Chouteau, P. O. Sassoulas, A. Vernhet, Y. Le Friec, F. Domengie, L. Scotti, D. Pacelli, J. L. Ogier, F. Boucard, S. Lagrasta, D. Benoit, L. Clement, P. Boivin, P. Ferreira, R. Annunziata, and P. Cappelletti, “Truly innovative 28nm fdsoi technology for automotive micro-controller applications embedding 16mb phase change memory,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [8] M. Barlas, A. Grossi, L. Grenouillet, E. Vianello, E. Nolot, N. Vaxelaire, P. Blaise, B. Traoré, J. Coignus, F. Perrin, R. Crochemore, F. Mazen, L. Lachal, S. Pauliac, C. Pellissier, S. Bernasconi, S. Chevalliez, J. F. Nodin, L. Perniola, and E. Nowak, “Improvement of hfo2 based rram array performances by local si implantation,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017\.
  * [9] S. Beamer, K. Asanovic, and D. Patterson, “Locality exists in graph processing: Workload characterization on an ivy bridge server,” in _IEEE International Symposium on Workload Characterization_ , 2015.
  * [10] A. Belmonte, J. Radhakrishnan, L. Goux, G. L. Donadio, P. Kumbhare, A. Redolfi, R. Delhougne, L. Nyns, W. Devulder, T. Witters, A. Covello, G. Vereecke, A. Franquet, V. Spampinato, S. Kundu, M. Mao, H. Hody, and G. S. Kar, “Co active electrode enhances cbram performance and scaling potential,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [11] E. M. Boujamaa, S. M. Ali, S. N. Wandji, A. Gourio, S. Pyo, G. Koh, Y. Song, T. Song, J. Kye, J. C. Vial, A. Sowden, M. Rathor, and C. Dray, “A 14.7mb/mm2 28nm fdsoi stt-mram with current starved read path, 52ohm/sigma offset voltage sense amplifier and fully trimmable ctat reference,” in _IEEE Symposium on VLSI Circuits_ , 2020.
  * [12] A. Bricalli, E. Ambrosi, M. Laudato, M. Maestro, R. Rodriguez, and D. Ielmini, “Siox-based resistive switching memory (rram) for crossbar storage/select elements with high on/off ratio,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [13] J. Bucek, K.-D. Lange, and J. v. Kistowski, “Spec cpu2017: Next-generation compute benchmark,” in _ACM/SPEC International Conference on Performance Engineering_ , 2018.
  * [14] R. Carboni, S. Ambrogio, W. Chen, M. Siddik, J. Harms, A. Lyle, W. Kula, G. Sandhu, and D. Ielmini, “Understanding cycling endurance in perpendicular spin-transfer torque (p-stt) magnetic memory,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [15] T. E. Carlson, W. Heirman, S. Eyerman, I. Hur, and L. Eeckhout, “An evaluation of high-level mechanistic core models,” _ACM Transactions on Architecture and Code Optimization (TACO)_ , 2014.
  * [16] C. Y. Chan, K. Y. Chen, H. K. Peng, and Y. H. Wu, “Fefet memory featuring large memory window and robust endurance of long-pulse cycling by interface engineering using high-k alon,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [17] T. Chang, Y. Chiu, C. Lee, J. Hung, K. Chang, C. Xue, S. Wu, H. Kao, P. Chen, H. Huang, S. Teng, and M. Chang, “13.4 a 22nm 1mb 1024b-read and near-memory-computing dual-mode stt-mram macro with 42.6gb/s read bandwidth for security-aware mobile devices,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020.
  * [18] A. Chen, “A review of emerging non-volatile memory (nvm) technologies and applications,” _Solid-State Electronics_ , 2016, extended papers selected from ESSDERC 2015.
  * [19] P. Chen, X. Peng, and S. Yu, “Neurosim+: An integrated device-to-algorithm framework for benchmarking synaptic devices and array architectures,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [20] W. Chen, W. Lin, L. Lai, S. Li, C. Hsu, H. Lin, H. Lee, J. Su, Y. Xie, S. Sheu, and M. Chang, “A 16mb dual-mode reram macro with sub-14ns computing-in-memory and memory functions enabled by self-write termination scheme,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [21] W. H. Chen, K. X. Li, W. Y. Lin, K. H. Hsu, P. Y. Li, C. H. Yang, C. X. Xue, E. Y. Yang, Y. K. Chen, Y. S. Chang, T. H. Hsu, Y. C. King, C. J. Lin, R. S. Liu, C. C. Hsieh, K. T. Tang, and M. F. Chang, “A 65nm 1mb nonvolatile computing-in-memory reram macro with sub-16ns multiply-and-accumulate for binary dnn ai edge processors,” in _IEEE International Solid - State Circuits Conference - (ISSCC)_ , 2018.
  * [22] H. Y. Cheng, W. C. Chien, I. T. Kuo, E. K. Lai, Y. Zhu, J. L. Jordan-Sweet, A. Ray, F. Carta, F. M. Lee, P. H. Tseng, M. H. Lee, Y. Y. Lin, W. Kim, R. Bruce, C. W. Yeh, C. H. Yang, M. BrightSky, and H. L. Lung, “An ultra high endurance and thermally stable selector based on teasgesise chalcogenides compatible with beol ic integration for cross-point pcm,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [23] H. Y. Cheng, I. T. Kuo, W. C. Chien, C. W. Yeh, Y. C. Chou, N. Gong, L. Gignac, C. H. Yang, C. W. Cheng, C. Lavoie, M. Hopstaken, R. L. Bruce, L. Buzi, E. K. Lai, F. Carta, A. Ray, M. H. Lee, H. Y. Ho, W. Kim, M. BrightSky, and H. L. Lung, “Si incorporation into assege chalcogenides for high thermal stability, high endurance and extremely low vth drift 3d stackable cross-point memory,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [24] P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, “Prime: A novel processing-in-memory architecture for neural network computation in reram-based main memory,” _ACM SIGARCH Computer Architecture News_ , 2016\.
  * [25] W. C. Chien, H. Y. Cheng, M. BrightSky, A. Ray, C. W. Yeh, W. Kim, R. Bruce, Y. Zhu, H. Y. Ho, H. L. Lung, and C. Lam, “Reliability study of a 128mb phase change memory chip implemented with doped gasbge with extraordinary thermal stability,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [26] Y. Chih, Y. Shih, C. Lee, Y. Chang, P. Lee, H. Lin, Y. Chen, C. Lo, M. Shih, K. Shen, H. Chuang, and T. J. Chang, “13.3 a 22nm 32mb embedded stt-mram with 10ns read speed, 1m cycle write endurance, 10 years retention at 150c and high immunity to magnetic field interference,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020\.
  * [27] Y.-C. Chiu, H.-W. Hu, L.-Y. Lai, T.-Y. Huang, H.-Y. Kao, K.-T. Chang, M.-S. Ho, C.-C. Chou, Y.-D. Chih, T.-Y. Chang, and M.-F. Chang, “A 40nm 2mb reram macro with 85% reduction in forming time and 99% reduction in page-write time using auto-forming and auto-write schemes,” in _Symposium on VLSI Technology_ , 2019.
  * [28] C. Chou, Z. Lin, C. Lai, C. Su, P. Tseng, W. Chen, W. Tsai, W. Chu, T. Ong, H. Chuang, Y. Chih, and T. Chang, “A 22nm 96kx144 rram macro with a self-tracking reference and a low ripple charge pump to achieve a configurable read window and a wide operating voltage range,” in _IEEE Symposium on VLSI Circuits_ , 2020.
  * [29] C. Chou, Z. Lin, P. Tseng, C. Li, C. Chang, W. Chen, Y. Chih, and T. J. Chang, “An n40 256k×44 embedded rram macro with sl-precharge sa and low-voltage current limiter to improve read and write performance,” in _IEEE International Solid - State Circuits Conference - (ISSCC)_ , 2018.
  * [30] T. Chou, W. Tang, J. Botimer, and Z. Zhang, “Cascade: Connecting rrams to extend analog dataflow in an end-to-end in-memory processing paradigm,” in _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_ , 2019.
  * [31] S. Chung, T. Kishi, J. W. Park, M. Yoshikawa, K. S. Park, T. Nagase, K. Sunouchi, H. Kanaya, G. C. Kim, K. Noma, M. S. Lee, A. Yamamoto, K. M. Rho, K. Tsuchida, S. J. Chung, J. Y. Yi, H. S. Kim, Y. S. Chun, H. Oyamatsu, and S. J. Hong, “4gbit density stt-mram using perpendicular mtj realized with compact cell structure,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [32] D. Datta, H. Dixit, S. Agarwal, A. Dasgupta, M. Tran, D. Houssameddine, Y. S. Chauhan, D. Shum, and F. Benistant, “Quantitative model for switching asymmetry in perpendicular mtj: A material-device-circuit co-design,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [33] S. Deng, G. Yin, W. Chakraborty, S. Dutta, S. Datta, X. Li, and K. Ni, “A comprehensive model for ferroelectric fet capturing the key behaviors: Scalability, variation, stochasticity, and accumulation,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [34] M. Donato, L. Pentecost, D. Brooks, and G. Wei, “Memti: Optimizing on-chip nonvolatile storage for visual multitask inference at the edge,” _IEEE Micro_ , 2019.
  * [35] M. Donato, B. Reagen, L. Pentecost, U. Gupta, D. Brooks, and G.-Y. Wei, “On-chip deep neural network storage with multi-level envm,” in _Proceedings of the 55th Annual Design Automation Conference_ , 2018.
  * [36] Q. Dong, Z. Wang, J. Lim, Y. Zhang, Y. Shih, Y. Chih, J. Chang, D. Blaauw, and D. Sylvester, “A 1mb 28nm stt-mram with 2.8ns read access time at 1.2v vdd using single-cap offset-cancelled sense amplifier and in-situ self-write-termination,” in _IEEE International Solid - State Circuits Conference - (ISSCC)_ , 2018.
  * [37] X. Dong, C. Xu, Y. Xie, and N. P. Jouppi, “Nvsim: A circuit-level performance, energy, and area model for emerging nonvolatile memory,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 2012.
  * [38] S. Dunkel, M. Trentzsch, R. Richter, P. Moll, C. Fuchs, O. Gehring, M. Majer, S. Wittek, B. Muller, T. Melde, H. Mulaosmanovic, S. Slesazeck, S. Müller, J. Ocker, M. Noack, D. . Löhr, P. Polakowski, J. Muller, T. Mikolajick, J. Hontschel, B. Rice, J. Pellerin, and S. Beyer, “A fefet based super-low-power ultra-fast embedded nvm technology for 22nm fdsoi and beyond,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [39] T. Endoh, H. Honjo, K. Nishioka, and S. Ikeda, “Recent progresses in stt-mram and sot-mram for next generation mram,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [40] H. Esmaeilzadeh, E. Blem, R. S. Amant, K. Sankaralingam, and D. Burger, “Dark silicon and the end of multicore scaling,” in _38th Annual International Symposium on Computer Architecture (ISCA)_ , 2011.
  * [41] X. Feng, Y. Li, L. Wang, Z. G. Yu, S. Chen, W. C. Tan, N. Macadam, G. Hu, X. Gong, T. Hasan, Y. W. Zhang, A. V. Y. Thean, and K. W. Ang, “First demonstration of a fully-printed mos2rram on flexible substrate with ultra-low switching voltage and its application as electronic synapse,” in _Symposium on VLSI Technology_ , 2019.
  * [42] K. Florent, M. Pesic, A. Subirats, K. Banerjee, S. Lavizzari, A. Arreghini, L. Di Piazza, G. Potoms, F. Sebaai, S. R. C. McMitchell, M. Popovici, G. Groeseneken, and J. Van Houdt, “Vertical ferroelectric hfo2 fet based on 3d nand architecture: Towards dense low-power memory,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [43] S. Fujii, Y. Kamimuta, T. Ino, Y. Nakasaki, R. Takaishi, and M. Saitoh, “First demonstration and performance improvement of ferroelectric hfo2-based resistive switch with low operation current and intrinsic diode property,” in _IEEE Symposium on VLSI Technology_ , 2016\.
  * [44] S. Fukami, T. Anekawa, A. Ohkawara, C. Zhang, and H. Ohno, “A sub-ns three-terminal spin-orbit torque induced switching device,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [45] W. J. Gallagher, E. Chien, T. Chiang, J. Huang, M. Shih, C. Y. Wang, C. Weng, S. Chen, C. Bair, G. Lee, Y. Shih, C. Lee, P. Lee, R. Wang, K. H. Shen, J. J. Wu, W. Wang, and H. Chuang, “22nm stt-mram for reflow and automotive uses with high yield, reliability, and magnetic immunity and with performance and shielding options,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [46] I. Giannopoulos, A. Sebastian, M. Le Gallo, V. P. Jonnalagadda, M. Sousa, M. N. Boon, and E. Eleftheriou, “8-bit precision in-memory multiplication with projected phase-change memory,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [47] O. Golonzka, J. Alzate, U. Arslan, M. Bohr, P. Bai, J. Brockman, B. Buford, C. Connor, N. Das, B. Doyle, T. Ghani, F. Hamzaoglu, P. Heil, P. Hentges, R. Jahan, D. Kencke, B. Lin, M. Lu, M. Mainuddin, M. Meterelliyoz, P. Nguyen, D. Nikonov, K. O’brien, J. O. Donnell, K. Oguz, D. Ouellette, J. Park, J. Pellegren, C. Puls, P. Quintero, T. Rahman, A. Romang, M. Sekhar, A. Selarka, M. Seth, A. J. Smith, A. K. Smith, L. Wei, C. Wiegand, Z. Zhang, and K. Fischer, “Mram as embedded non-volatile memory solution for 22ffl finfet technology,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [48] O. Golonzka, U. Arslan, P. Bai, M. Bohr, O. Baykan, Y. Chang, A. Chaudhari, A. Chen, J. Clarke, C. Connor, N. Das, C. English, T. Ghani, F. Hamzaoglu, P. Hentges, P. Jain, C. Jezewski, I. Karpov, H. Kothari, R. Kotlyar, B. Lin, M. Metz, J. Odonnell, D. Ouellette, J. Park, A. Pirkle, P. Quintero, D. Seghete, M. Sekhar, A. S. Gupta, M. Seth, N. Strutt, C. Wiegand, H. J. Yoo, and K. Fischer, “Non-volatile rram embedded into 22ffl finfet technology,” in _Symposium on VLSI Technology_ , 2019.
  * [49] O. Golonzka, U. Arslan, P. Bai, M. Bohr, O. Baykan, Y. Chang, A. Chaudhari, A. Chen, J. Clarke, C. Connor, N. Das, C. English, T. Ghani, F. Hamzaoglu, P. Hentges, P. Jain, C. Jezewski, I. Karpov, H. Kothari, R. Kotlyar, B. Lin, M. Metz, J. Odonnell, D. Ouellette, J. Park, A. Pirkle, P. Quintero, D. Seghete, M. Sekhar, A. S. Gupta, M. Seth, N. Strutt, C. Wiegand, H. J. Yoo, and K. Fischer, “Non-volatile rram embedded into 22ffl finfet technology,” in _Symposium on VLSI Technology_ , 2019.
  * [50] N. Gong, W. Chien, Y. Chou, C. Yeh, N. Li, H. Cheng, C. Cheng, I. Kuo, C. Yang, R. Bruce, A. Ray, L. Gignac, Y. Lin, C. Miller, T. Perri, W. Kim, L. Buzi, H. Utomo, F. Carta, E. Lai, H. Ho, H. Lung, and M. BrightSky, “A no-verification multi-level-cell (mlc) operation in cross-point ots-pcm,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [51] L. Goux, A. Belmonte, U. Celano, J. Woo, S. Folkersma, C. Y. Chen, A. Redolfi, A. Fantini, R. Degraeve, S. Clima, W. Vandervorst, and M. Jurczak, “Retention, disturb and variability improvements enabled by local chemical-potential tuning and controlled hour-glass filament shape in a novel wwo3al2o3cu cbram,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [52] T. J. Ham, L. Wu, N. Sundaram, N. Satish, and M. Martonosi, “Graphicionado: A high-performance and energy-efficient accelerator for graph analytics,” in _49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2016.
  * [53] A. Hankin, T. Shapira, K. Sangaiah, M. Lui, and M. Hempstead, “Evaluation of non-volatile memory based last level cache given modern use case behavior,” in _IEEE International Symposium on Workload Characterization (IISWC)_ , 2019.
  * [54] Hewlett Packard, “Cacti 7.0,” 2017.
  * [55] C. Ho, S. Chang, C. Huang, Y. Chuang, S. Lim, M. Hsieh, S. Chang, and H. Liao, “Integrated hfo2-rram to achieve highly reliable, greener, faster, cost-effective, and scaled devices,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [56] H. Honjo, T. V. A. Nguyen, T. Watanabe, T. Nasuno, C. Zhang, T. Tanigawa, S. Miura, H. Inoue, M. Niwa, T. Yoshiduka, Y. Noguchi, M. Yasuhira, A. Tamakoshi, M. Natsui, Y. Ma, H. Koike, Y. Takahashi, K. Furuya, H. Shen, S. Fukami, H. Sato, S. Ikeda, T. Hanyu, H. Ohno, and T. Endoh, “First demonstration of field-free sot-mram with 0.35 ns write speed and 70 thermal stability under 400c thermal tolerance by canted sot structure and its advanced patterning/sot channel technology,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [57] E. Hsieh, X. Zheng, M. Nelson, B. Le, H. Wong, S. Mitra, S. Wong, M. Giordano, B. Hodson, A. Levy, S. Osekowsky, R. Radway, Y. Shih, W. Wan, and T. Wu, “High-density multiple bits-per-cell 1t4r rram array with gradual set/reset and its effectiveness for deep learning,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [58] F. K. Hsueh, C. H. Shen, J. M. Shieh, K. S. Li, H. C. Chen, W. H. Huang, H. H. Wang, C. C. Yang, T. Y. Hsieh, C. H. Lin, B. Y. Chen, Y. S. Shiao, G. W. Huang, O. Y. Wong, P. H. Chen, and W. K. Yeh, “First fully functionalized monolithic 3d iot chip with 0.5 v light-electricity power management, 6.8 ghz wireless-communication vco, and 4-layer vertical reram,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [59] G. Hu, M. G. Gottwald, Q. He, J. H. Park, G. Lauer, J. J. Nowak, S. L. Brown, B. Doris, D. Edelstein, E. R. Evarts, P. Hashemi, B. Khan, Y. H. Kim, C. Kothandaraman, N. Marchack, E. J. O’Sullivan, M. Reuter, R. P. Robertazzi, J. Z. Sun, T. Suwannasiri, P. L. Trouilloud, Y. Zhu, and D. C. Worledge, “Key parameters affecting stt-mram switching efficiency and improved device performance of 400c-compatible p-mtjs,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [60] G. Hu, J. J. Nowak, M. G. Gottwald, S. L. Brown, B. Doris, C. P. D’Emic, P. Hashemi, D. Houssameddine, Q. He, D. Kim, J. Kim, C. Kothandaraman, G. Lauer, H. K. Lee, N. Marchack, M. Reuter, R. P. Robertazzi, J. Z. Sun, T. Suwannasiri, P. L. Trouilloud, S. Woo, and D. C. Worledge, “Spin-transfer torque mram with reliable 2 ns writing for last level cache applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019.
  * [61] IEEE, “International roadmap for devices and systems (IRDS) 2020 edition.”
  * [62] A. Inci, M. M. Isgenc, and D. Marculescu, “DeepNVM++: cross-layer modeling and optimization framework of non-volatile memories for deep learning,” _preprint arXiv:2012.04559_ , 2020.
  * [63] A. F. Inci, M. Meric Isgenc, and D. Marculescu, “Deepnvm: A framework for modeling and analysis of non-volatile memory technologies for deep learning applications,” in _Design, Automation Test in Europe Conference Exhibition (DATE)_ , 2020.
  * [64] P. Jain, U. Arslan, M. Sekhar, B. C. Lin, L. Wei, T. Sahu, J. Alzate-vinasco, A. Vangapaty, M. Meterelliyoz, N. Strutt, A. B. Chen, P. Hentges, P. A. Quintero, C. Connor, O. Golonzka, K. Fischer, and F. Hamzaoglu, “13.2 a 3.6mb 10.1mb/mm2 embedded non-volatile reram macro in 22nm finfet technology with adaptive forming/set/reset schemes yielding down to 0.5v with sensing time of 5ns at 0.7v,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2019.
  * [65] G. Jan, L. Thomas, S. Le, Y. Lee, H. Liu, J. Zhu, J. Iwata-Harms, S. Patel, R. Tong, S. Serrano-Guisan, D. Shen, R. He, J. Haq, J. Teng, V. Lam, R. Annapragada, Y. Wang, T. Zhong, T. Torng, and P. Wang, “Achieving sub-ns switching of stt-mram for future embedded llc applications through improvement of nucleation and propagation switching mechanisms,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [66] M. Jerry, P. Chen, J. Zhang, P. Sharma, K. Ni, S. Yu, and S. Datta, “Ferroelectric fet analog synapse for acceleration of deep neural network training,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [67] Z. Jiang, Z. Wang, X. Zheng, S. Fong, S. Qin, H. Y. Chen, C. Ahn, J. Cao, Y. Nishi, and H. . S. P. Wong, “Microsecond transient thermal behavior of hfox-based resistive random access memory using a micro thermal stage (mts),” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016\.
  * [68] J. J. Kan, C. Park, C. Ching, J. Ahn, L. Xue, R. Wang, A. Kontos, S. Liang, M. Bangar, H. Chen, S. Hassan, S. Kim, M. Pakala, and S. H. Kang, “Systematic validation of 2x nm diameter perpendicular mtj arrays and mgo barrier for sub-10 nm embedded stt-mram with practically unlimited endurance,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [69] F. Khan, D. Moy, D. Anand, E. H. Schroeder, R. Katz, L. Jiang, E. Banghart, N. Robson, and T. Kirihata, “Turning logic transistors into secure, multi-time programmable, embedded non-volatile memory elements for 14 nm finfet technologies and beyond,” in _Symposium on VLSI Technology_ , 2019.
  * [70] W. S. Khwa, M. F. Chang, J. Y. Wu, M. H. Lee, T. H. Su, K. H. Yang, T. F. Chen, T. Y. Wang, H. P. Li, M. BrightSky, S. Kim, H. L. Lung, and C. Lam, “7.3 a resistance-drift compensation scheme to reduce mlc pcm raw ber by over 100× for storage-class memory applications,” in _IEEE International Solid-State Circuits Conference (ISSCC)_ , 2016.
  * [71] S. G. Kim, J. C. Lee, T. J. Ha, J. H. Lee, J. Y. Lee, Y. T. Park, K. W. Kim, W. K. Ju, Y. S. Ko, H. M. Hwang, B. M. Lee, J. Y. Moon, W. Y. Park, B. G. Gyun, B. Lee, D. Yim, and S. Hong, “Breakthrough of selector technology for cross-point 25-nm reram,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [72] W. Kim, A. Hardtdegen, C. Rodenbücher, S. Menzel, D. J. Wouters, S. Hoffmann-Eifert, D. Buca, R. Waser, and V. Rana, “Forming-free metal-oxide reram by oxygen ion implantation process,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [73] M. Kobayashi, N. Ueyama, K. Jang, and T. Hiramoto, “Experimental study on polarization-limited operation speed of negative capacitance fet with ferroelectric hfo2,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [74] K. Korgaonkar, I. Bhati, H. Liu, J. Gaur, S. Manipatruni, S. Subramoney, T. Karnik, S. Swanson, I. Young, and H. Wang, “Density tradeoffs of non-volatile memory as a replacement for sram based last level cache,” in _ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)_ , 2018.
  * [75] Z. Krivokapic, U. Rana, R. Galatage, A. Razavieh, A. Aziz, J. Liu, J. Shi, H. J. Kim, R. Sporer, C. Serrao, A. Busquet, P. Polakowski, J. Müller, W. Kleemeier, A. Jacob, D. Brown, A. Knorr, R. Carter, and S. Banna, “14nm ferroelectric finfet technology with steep subthreshold slope for ultra low power applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [76] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” _ArXiv_ , vol. abs/1909.11942, 2020.
  * [77] K. Lee, J. H. Bak, Y. J. Kim, C. K. Kim, A. Antonyan, D. H. Chang, S. H. Hwang, G. W. Lee, N. Y. Ji, W. J. Kim, J. H. Lee, B. J. Bae, J. H. Park, I. H. Kim, B. Y. Seo, S. H. Han, Y. Ji, H. T. Jung, S. O. Park, O. I. Kwon, J. W. Kye, Y. D. Kim, S. W. Pae, Y. J. Song, G. T. Jeong, K. H. Hwang, G. H. Koh, H. K. Kang, and E. S. Jung, “1gbit high density embedded stt-mram in 28nm fdsoi technology,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2019\.
  * [78] K. Lee, R. Chao, K. Yamane, V. B. Naik, H. Yang, J. Kwon, N. L. Chung, S. H. Jang, B. Behin-Aein, J. H. Lim, S. K, B. Liu, E. H. Toh, K. W. Gan, D. Zeng, N. Thiyagarajah, L. C. Goh, T. Ling, J. W. Ting, J. Hwang, L. Zhang, R. Low, R. Krishnan, L. Zhang, S. L. Tan, Y. S. You, C. S. Seet, H. Cong, J. Wong, S. T. Woo, E. Quek, and S. Y. Siah, “22-nm fd-soi embedded mram technology for low-power automotive-grade-l mcu applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [79] M. H. Lee, P. Chen, S. Fan, Y. Chou, C.Kuo, C. Tang, H. Chen, S. Gu, R. Hong, Z. Wang, S. Chen, C. Liao, K. Chen, S. T. Chang, M. Liao, K. Li, and C. W. Liu, “Ferroelectric al:hfo2 negative capacitance fets,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [80] M. H. Lee, S. Fan, C. Tang, P. Chen, Y. Chou, H. Chen, J. Kuo, M. Xie, S. Liu, M. Liao, C. Jong, K. Li, M. Chen, and C. W. Liu, “Physical thickness 1.x nm ferroelectric hfzrox negative capacitance fets,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [81] T. Lee, K. Yamane, J. Kwon, V. Naik, Y. Otani, D. Zeng, J. Lim, K. Sivabalan, C. Chiang, Y. Huang, S. Jang, L. Hau, R. Chao, N. Chung, W. Neo, K. Khua, N. Thiyagarajah, T. Ling, L. Goh, and S. Siah, “Fast switching of stt-mram to realize high speed applications,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [82] Y. Lee, Y. Song, J. Kim, S. Oh, B.-J. Bae, S. Lee, J. Lee, U. Pi, B. Seo, H. Jung, K. Lee, H. Shin, H. Jung, M. Pyo, A. Antonyan, D. Lee, S. Hwang, D. Jang, Y. Ji, and E. Jung, “Embedded stt-mram in 28-nm fdsoi logic process for industrial mcu/iot application,” in _Symposium on VLSI Technology_ , 2018\.
  * [83] J. Leskovec and A. Krevl, “SNAP Datasets: Stanford large network dataset collection,” June 2014.
  * [84] H. Li, K. S. Li, C. H. Lin, J. L. Hsu, W. C. Chiu, M. C. Chen, T. T. Wu, J. Sohn, S. B. Eryilmaz, J. M. Shieh, W. K. Yeh, and H. P. Wong, “Four-layer 3d vertical rram integrated with finfet as a versatile computing unit for brain-inspired cognitive information processing,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [85] H. Li, T. F. Wu, A. Rahimi, K. S. Li, M. Rusch, C. H. Lin, J. L. Hsu, M. M. Sabry, S. B. Eryilmaz, J. Sohn, W. C. Chiu, M. C. Chen, T. T. Wu, J. M. Shieh, W. K. Yeh, J. M. Rabaey, S. Mitra, and H. . S. P. Wong, “Hyperdimensional computing with 3d vrram in-memory kernels: Device-architecture co-design for energy-efficient, error-resilient language recognition,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [86] C. Lin, J. Hung, W. Lin, C. Lo, Y. Chiang, H. Tsai, G. Yang, Y. King, C. J. Lin, T. Chen, and M. Chang, “7.4 a 256b-word length reram-based tcam with 1ns search-time and 14x improvement in word length-energy efficiency-density product using 2.5t1r cell,” in _IEEE International Solid-State Circuits Conference (ISSCC)_ , 2016.
  * [87] C. S. Lin, W. T. Huang, A. Huang, Y. H. Yang, Y. L. Hsu, S. K. Yang, J. Liu, H. W. Tseng, J. Huang, B. Y. Chou, K. Huang, W. K. Chang, D. Chang, C. H. Chien, H. Yeh, P. W. Liu, C. D. Hsieh, H. Chuang, and A. Kalnitsky, “An approach to embedding traditional non-volatile memories into a deep sub-micron cmos,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [88] Y. H. Lin, Y. H. Ho, M. H. Lee, C. H. Wang, Y. Y. Lin, F. M. Lee, K. C. Hsu, P. H. Tseng, D. Y. Lee, K. H. Chiang, K. C. Wang, T. Y. Tseng, and C. Y. Lu, “A comprehensive study of 3-stage high resistance state retention behavior for tmo rerams from single cells to a large array,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [89] Q. Liu, B. Gao, P. Yao, D. Wu, J. Chen, Y. Pang, W. Zhang, Y. Liao, C. Xue, W. Chen, J. Tang, Y. Wang, M. Chang, H. Qian, and H. Wu, “33.2 a fully integrated analog reram based 78.4tops/w compute-in-memory chip with fully parallel mac computing,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020.
  * [90] Y. Liu, Z. Wang, A. Lee, F. Su, C. P. Lo, Z. Yuan, C. C. Lin, Q. Wei, Y. Wang, Y. C. King, C. J. Lin, P. Khalili, K. L. Wang, M. F. Chang, and H. Yang, “4.7 a 65nm reram-enabled nonvolatile processor with 6x reduction in restore time and 4x higher clock frequency using adaptive data retention and self-write-termination nonvolatile logic,” in _IEEE International Solid-State Circuits Conference (ISSCC)_ , 2016.
  * [91] C. P. Lo, W. H. Chen, Z. Wang, A. Lee, K. H. Hsu, F. Su, Y. C. King, C. J. Lin, Y. Liu, H. Yang, P. Khalili, K. L. Wang, and M. F. Chang, “A reram-based single-nvm nonvolatile flip-flop with reduced stress-time and write-power against wide distribution in write-time by using self-write-termination scheme for nonvolatile processors in iot era,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [92] H. L. Lung, Y. H. Ho, Y. Zhu, W. C. Chien, S. Kim, W. Kim, H. Y. Cheng, A. Ray, M. Brightsky, R. Bruce, C. W. Yeh, and C. Lam, “A novel low power phase change memory using inter-granular switching,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [93] Q. Luo, X. Xu, T. Gong, H. Lv, D. Dong, H. Ma, P. Yuan, J. Gao, J. Liu, Z. Yu, J. Li, S. Long, Q. Liu, and M. Liu, “8-layers 3d vertical rram with excellent scalability towards storage class memory applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017\.
  * [94] H. Lv, X. Xu, P. Yuan, D. Dong, T. Gong, J. Liu, Z. Yu, P. Huang, K. Zhang, C. Huo, C. Chen, Y. Xie, Q. Luo, S. Long, Q. Liu, J. Kang, D. Yang, S. Yin, S. Chiu, and M. Liu, “Beol based rram with one extra-mask for low cost, highly reliable embedded application in 28 nm node and beyond,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [95] J. Ma, Z. Chai, W. Zhang, B. Govoreanu, J. F. Zhang, Z. Ji, B. Benbakhti, G. Groeseneken, and M. Jurczak, “Identify the critical regions and switching/failure mechanisms in non-filamentary rram (a-vmco) by rtn and cvs techniques for memory window improvement,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [96] V. Milo, G. Pedretti, R. Carboni, A. Calderoni, N. Ramaswamy, S. Ambrogio, and D. Ielmini, “Demonstration of hybrid cmos/rram neural networks with spike time/rate-dependent plasticity,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [97] S. Miura, K. Nishioka, H. Naganuma, T. V. A. Nguyen, H. Honjo, S. Ikeda, T. Watanabe, H. Inoue, M. Niwa, T. Tanigawa, Y. Noguchi, T. Yoshiduka, M. Yasuhira, and T. Endoh, “Scalability of quad interface p-mtj for 1x nm stt-mram with 10 ns low power write operation, 10 years retention and endurance 1011,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [98] F. Mo, Y. Tagawa, C. Jin, M. Ahn, T. Saraya, T. Hiramoto, and M. Kobayashi, “Experimental demonstration of ferroelectric hfo2 fet with ultrathin-body igzo for high-density and low-power memory application,” in _Symposium on VLSI Technology_ , 2019.
  * [99] H. Mulaosmanovic, J. Ocker, S. Müller, M. Noack, J. Müller, P. Polakowski, T. Mikolajick, and S. Slesazeck, “Novel ferroelectric fet based synapse for neuromorphic systems,” in _Symposium on VLSI Technology_ , 2017.
  * [100] V. B. Naik, J. H. Lim, K. Yamane, D. Zeng, H. Yang, N. Thiyagarajah, J. Kwon, N. L. Chung, R. Chao, T. Ling, and K. Lee, “Superior endurance performance of 22-nm embedded mram technology,” in _IEEE International Reliability Physics Symposium (IRPS)_ , 2019.
  * [101] V. B. Naik, K. Yamane, J. H. Lim, T. Y. Lee, J. Kwon, B. Aein, N. L. Chung, L. Y. Hau, R. Chao, D. Zeng, Y. Otani, C. Chiang, Y. Huang, L. Pu, N. Thiyagarajah, S. H. Jang, W. P. Neo, H. Dixit, S. K. Aris, L. C. Goh, T. Ling, J. Hwang, J. W. Ting, L. Zhang, R. Low, N. Balasankaran, C. S. Seet, S. Ong, J. Wong, Y. S. You, S. T. Woo, and S. Y. Siah, “A reliable tddb lifetime projection model verified using 40mb stt-mram macro at sub-ppm failure rate to realize unlimited endurance for cache applications,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [102] C. Nail, G. Molas, P. Blaise, G. Piccolboni, B. Sklenard, C. Cagli, M. Bernard, A. Roule, M. Azzaz, E. Vianello, C. Carabasse, R. Berthier, D. Cooper, C. Pelissier, T. Magis, G. Ghibaudo, C. Vallée, D. Bedeau, O. Mosendz, B. De Salvo, and L. Perniola, “Understanding rram endurance, retention and window margin trade-off using experimental results and simulations,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [103] M. Natsui, A. Tamakoshi, H. Honjo, T. Watanabe, T. Nasuno, C. Zhang, T. Tanigawa, H. Inoue, M. Niwa, T. Yoshiduka, Y. Noguchi, M. Yasuhira, Y. Ma, H. Shen, S. Fukami, H. Sato, S. Ikeda, H. Ohno, T. Endoh, and T. Hanyu, “Dual-port field-free sot-mram achieving 90-mhz read and 60-mhz write operations under 55-nm cmos technology and 1.2-v supply voltage,” in _IEEE Symposium on VLSI Circuits_ , 2020.
  * [104] V. D. Nguyen, P. Sabon, J. Chatterjee, L. Tille, P. V. Coelho, S. Auffret, R. Sousa, L. Prejbeanu, E. Gautier, L. Vila, and B. Dieny, “Novel approach for nano-patterning magnetic tunnel junctions stacks at narrow pitch: A route towards high density stt-mram applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [105] K. Ni, W. Chakraborty, J. Smith, B. Grisafe, and S. Datta, “Fundamental understanding and control of device-to-device variation in deeply scaled ferroelectric fets,” in _Symposium on VLSI Technology_ , 2019\.
  * [106] H. Noguchi, K. Ikegami, K. Abe, S. Fujita, Y. Shiota, T. Nozaki, S. Yuasa, and Y. Suzuki, “Novel voltage controlled mram (vcm) with fast read/write circuits for ultra large last level cache,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [107] H. Noguchi, K. Ikegami, S. Takaya, E. Arima, K. Kushida, A. Kawasumi, H. Hara, K. Abe, N. Shimomura, J. Ito, S. Fujita, T. Nakada, and H. Nakamura, “7.2 4mb stt-mram-based cache with memory-access-aware power optimization and write-verify-write read-modify-write scheme,” in _IEEE International Solid-State Circuits Conference (ISSCC)_ , 2016.
  * [108] NVIDIA, “Nvidia deep learning accelerator (nvdla): a free and open architecture that promotes a standard way to design deep learning inference accelerators. nvdla.org,” 2017.
  * [109] J. Okuno, T. Kunihiro, K. Konishi, H. Maemura, Y. Shuto, F. Sugaya, M. Materano, T. Ali, K. Kuehnel, K. Seidel, U. Schroeder, T. Mikolajick, M. Tsukamoto, and T. Umebayashi, “Soc compatible 1t1c feram memory array based on ferroelectric hf05zr05o2,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [110] Y. Pang, H. Wu, B. Gao, D. Wu, A. Chen, and H. Qian, “A novel puf against machine learning attack: Implementation on a 16 mb rram chip,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [111] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , 2019\.
  * [112] L. Pentecost, M. Donato, B. Reagen, U. Gupta, S. Ma, G.-Y. Wei, and D. Brooks, “Maxnvm: Maximizing dnn storage density and inference efficiency with sparse encoding and error mitigation,” in _Proceedings of the 52nd International Symposium on Microarchitecture_ , 2019.
  * [113] M. Poremba and Y. Xie, “Nvmain: An architectural-level main memory simulator for emerging non-volatile memories,” in _IEEE Computer Society Annual Symposium on VLSI_ , 2012.
  * [114] M. Poremba, S. Mittal, D. Li, J. Vetter, and Y. Xie, “Destiny: A tool for modeling emerging 3d nvm and edram caches,” in _Design, Automation Test in Europe Conference Exhibition (DATE)_ , 2015.
  * [115] F. M. Puglisi, L. Larcher, C. Pan, N. Xiao, Y. Shi, F. Hui, and M. Lanza, “2d hbn based rram devices,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [116] B. Reagen, U. Gupta, L. Pentecost, P. Whatmough, S. K. Lee, N. Mulholland, D. Brooks, and G. Wei, “Ares: A framework for quantifying the resilience of deep neural networks,” in _55th Design Automation Conference (DAC)_ , 2018.
  * [117] D. Saida, S. Kashiwada, M. Yakabe, T. Daibou, N. Hase, M. Fukumoto, S. Miwa, Y. Suzuki, H. Noguchi, S. Fujita, and J. Ito, “Sub-3 ns pulse with sub-100 microa switching of 1x–2x nm perpendicular mtj for high-performance embedded stt-mram towards sub-20 nm cmos,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [118] H. Sato, H. Honjo, T. Watanabe, M. Niwa, H. Koike, S. Miura, T. Saito, H. Inoue, T. Nasuno, T. Tanigawa, Y. Noguchi, T. Yoshiduka, M. Yasuhira, S. Ikeda, S. Y. Kang, T. Kubo, K. Yamashita, Y. Yagi, R. Tamura, and T. Endoh, “14ns write speed 128mb density embedded stt-mram with endurance 1010 and 10yrs retention 85c using novel low damage mtj integration process,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [119] A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,” _ACM SIGARCH Computer Architecture News_ , 2016.
  * [120] M. M. Sharifi, L. Pentecost, R. Rajaei, A. Kazemi, Q. Lou, G.-Y. Wei, D. Brooks, K. Ni, X. S. Hu, M. Niemier, and M. Donato, “Application-driven design exploration for dense ferroelectric embedded non-volatile memories,” in _Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED ’21)_ , 2021.
  * [121] A. A. Sharma, B. Doyle, H. J. Yoo, I. C. Tung, J. Kavalieros, M. V. Metz, M. Reshotko, P. Majhi, T. Brown-Heft, Y. J. Chen, and V. H. Le, “High speed memory operation in channel-last, back-gated ferroelectric transistors,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2020\.
  * [122] S. Shibayama, L. Xu, S. Migita, and A. Toriumi, “Study of wake-up and fatigue properties in doped and undoped ferroelectric hfo2 in conjunction with piezo-response force microscopy analysis,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [123] M. Shih, C. Wang, Y. Lee, W. Wang, L. Thomas, H. Liu, J. Zhu, Y. Lee, G. Jan, Y. Wang, T. Zhong, T. Torng, P. Wang, D. Lin, T. Chiang, K. Shen, H. Chuang, and W. J. Gallagher, “Reliability study of perpendicular stt-mram as emerging embedded memory qualified for reflow soldering at 260c,” in _IEEE Symposium on VLSI Technology_ , 2016\.
  * [124] Y. Shih, C. Lee, Y. Chang, P. Lee, H. Lin, Y. Chen, K. Lin, T. Yeh, H. Yu, H. Chuang, Y. Chih, and J. Chang, “Logic process compatible 40nm 16mb, embedded perpendicular-mram with hybrid-resistance reference, sub-microa sensing resolution, and 17.5ns read access time,” in _IEEE Symposium on VLSI Circuits_ , 2018.
  * [125] H. Shin, J. Kim, S. Kang, and S. Kwak, “A 28nm 10mb embedded flash memory for iot product with ultra-low power near-1v supply voltage and high temperature for grade 1 operation,” in _IEEE Symposium on VLSI Circuits_ , 2020.
  * [126] D. Shum, D. Houssameddine, S. T. Woo, Y. S. You, J. Wong, K. W. Wong, C. C. Wang, K. H. Lee, K. Yamane, V. B. Naik, C. S. Seet, T. Tahmasebi, C. Hai, H. W. Yang, N. Thiyagarajah, R. Chao, J. W. Ting, N. L. Chung, T. Ling, T. H. Chan, S. Y. Siah, R. Nair, S. Deshpande, R. Whig, K. Nagel, S. Aggarwal, M. DeHerrera, J. Janesky, M. Lin, H. Chia, M. Hossain, H. Lu, S. Ikegawa, F. B. Mancoff, G. Shimon, J. M. Slaughter, J. J. Sun, M. Tran, S. M. Alam, and T. Andre, “Cmos-embedded stt-mram arrays in 2x nm nodes for gp-mcu applications,” in _Symposium on VLSI Technology_ , 2017.
  * [127] F. Sijstermans, “The nvidia deep learning accelerator,” in _Hot Chips_ , 2018\.
  * [128] J. M. Slaughter, K. Nagel, R. Whig, S. Deshpande, S. Aggarwal, M. DeHerrera, J. Janesky, M. Lin, H. J. Chia, M. Hossain, S. Ikegawa, F. B. Mancoff, G. Shimon, J. J. Sun, M. Tran, T. Andre, S. M. Alam, F. Poh, J. H. Lee, Y. T. Chow, Y. Jiang, H. X. Liu, C. C. Wang, S. M. Noh, T. Tahmasebi, S. K. Ye, and D. Shum, “Technology for reliable spin-torque mram products,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [129] L. Song, X. Qian, H. Li, and Y. Chen, “Pipelayer: A pipelined reram-based accelerator for deep learning,” in _IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2017.
  * [130] Y. J. Song, J. H. Lee, S. H. Han, H. C. Shin, K. H. Lee, K. Suh, D. E. Jeong, G. H. Koh, S. C. Oh, J. H. Park, S. O. Park, B. J. Bae, O. I. Kwon, K. H. Hwang, B. Y. Seo, Y. K. Lee, S. H. Hwang, D. S. Lee, Y. Ji, K. C. Park, G. T. Jeong, H. S. Hong, K. P. Lee, H. K. Kang, and E. S. Jung, “Demonstration of highly manufacturable stt-mram embedded in 28nm logic,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [131] Y. J. Song, J. H. Lee, H. C. Shin, K. H. Lee, K. Suh, J. R. Kang, S. S. Pyo, H. T. Jung, S. H. Hwang, G. H. Koh, S. C. Oh, S. O. Park, J. K. Kim, J. C. Park, J. Kim, K. H. Hwang, G. T. Jeong, K. P. Lee, and E. S. Jung, “Highly functional and reliable 8mb stt-mram embedded in 28nm logic,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [132] Z. T. Song, D. L. Cai, X. Li, L. Wang, Y. F. Chen, H. P. Chen, Q. Wang, Y. P. Zhan, and M. H. Ji, “High endurance phase change memory chip implemented based on carbon-doped ge2sb2te5 in 40 nm node for embedded application,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018\.
  * [133] Stanford Nanoelectronics Lab, “Stanford memory trends.”
  * [134] F. Su, W. H. Chen, L. Xia, C. P. Lo, T. Tang, Z. Wang, K. H. Hsu, M. Cheng, J. Y. Li, Y. Xie, Y. Wang, M. F. Chang, H. Yang, and Y. Liu, “A 462gops/j rram-based nonvolatile intelligent processor for energy harvesting ioe system featuring nonvolatile logics and processing-in-memory,” in _Symposium on VLSI Technology_ , 2017.
  * [135] X. Sun, P. Wang, K. Ni, S. Datta, and S. Yu, “Exploiting hybrid precision for training and inference: A 2t-1fefet based analog synaptic weight cell,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018\.
  * [136] A. J. Tan, M. Pesic, L. Larcher, Y. H. Liao, L. C. Wang, J. H. Bae, C. Hu, and S. Salahuddin, “Hot electrons as the dominant source of degradation for sub-5nm hzo fefets,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [137] L. Thomas, G. Jan, S. Le, S. Serrano-Guisan, Y. Lee, H. Liu, J. Zhu, J. Iwata-Harms, R. Tong, S. Patel, V. Sundar, D. Shen, Y. Yang, R. He, J. Haq, Z. Teng, V. Lam, P. Liu, Y. Wang, T. Zhong, and P. Wang, “Probing magnetic properties of stt-mram devices down to sub-20 nm using spin-torque fmr,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [138] L. Thomas, G. Jan, S. Serrano-Guisan, H. Liu, J. Zhu, Y. Lee, S. Le, J. Iwata-Harms, R. Tong, S. Patel, V. Sundar, D. Shen, Y. Yang, R. He, J. Haq, Z. Teng, V. Lam, P. Liu, Y. Wang, T. Zhong, H. Fukuzawa, and P. Wang, “Stt-mram devices with low damping and moment optimized for llc applications at ox nodes,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [139] L. Tillie, E. Nowak, R. C. Sousa, M. Cyrille, B. Delaet, T. Magis, A. Persico, J. Langer, B. Ocker, I. Prejbeanu, and L. Perniola, “Data retention extraction methodology for perpendicular stt-mram,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [140] M. Trentzsch, S. Flachowsky, R. Richter, J. Paul, B. Reimer, D. Utess, S. Jansen, H. Mulaosmanovic, S. Müller, S. Slesazeck, J. Ocker, M. Noack, J. Müller, P. Polakowski, J. Schreiter, S. Beyer, T. Mikolajick, and B. Rice, “A 28nm hkmg super low power embedded nvm technology based on ferroelectric fets,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [141] W. Wan, R. Kubendran, S. B. Eryilmaz, W. Zhang, Y. Liao, D. Wu, S. Deiss, B. Gao, P. Raina, S. Joshi, H. Wu, G. Cauwenberghs, and H. P. Wong, “33.1 a 74 tmacs/w cmos-rram neurosynaptic core with dynamically reconfigurable dataflow and in-situ transposable weights for probabilistic graphical models,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020.
  * [142] C. Wang, C. McClellan, Y. Shi, X. Zheng, V. Chen, M. Lanza, E. Pop, and H. P. Wong, “3d monolithic stacked 1t1r cells using monolayer mos2 fet and hbn rram fabricated at low (150c) temperature,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [143] C. Wang, M. Shih, Y. Lee, W. Wang, L. Thomas, Y. Lee, H. Liu, J. Zhu, G. Jan, A. Wang, T. Zhong, P. Wang, D. Lin, C. Chen, C. Chang, C. Weng, T. Chiang, K. Shen, W. J. Gallagher, and H. Chuang, “Impact of external magnetic field on embedded perpendicular stt-mram technology qualified for solder reflow,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [144] C. Y. Wang, M. C. Shih, C. H. Weng, C. H. Chen, C. Y. Chang, W. Wang, T. W. Chiang, A. Hung, H. Chuang, and W. J. Gallagher, “Reliability demonstration of reflow qualified 22nm stt-mram for embedded memory applications,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [145] Z. Wang, Z. Li, L. Xu, Q. Dong, C. I. Su, W. T. Chu, G. Tsou, Y. D. Chih, T. Y. J. Chang, D. Sylvester, H. S. Kim, and D. Blaauw, “An all-weights-on-chip dnn accelerator in 22nm ull featuring 24×1 mb erram,” in _2020 IEEE Symposium on VLSI Circuits_.
  * [146] L. Wei, J. G. Alzate, U. Arslan, J. Brockman, N. Das, K. Fischer, T. Ghani, O. Golonzka, P. Hentges, R. Jahan, P. Jain, B. Lin, M. Meterelliyoz, J. O’Donnell, C. Puls, P. Quintero, T. Sahu, M. Sekhar, A. Vangapaty, C. Wiegand, and F. Hamzaoglu, “13.3 a 7mb stt-mram in 22ffl finfet technology with 4ns read sensing time at 0.9v using write-verify-write scheme and offset-cancellation sensing technique,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2019.
  * [147] Z. Wei, Y. Katoh, S. Ogasahara, Y. Yoshimoto, K. Kawai, Y. Ikeda, K. Eriguchi, K. Ohmori, and S. Yoneda, “True random number generator using current difference based on a fractional stochastic model in 40-nm embedded reram,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [148] H. Wu, P. Yao, B. Gao, W. Wu, Q. Zhang, W. Zhang, N. Deng, D. Wu, H. P. Wong, S. Yu, and H. Qian, “Device and circuit optimization of rram for neuromorphic computing,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [149] J. Wu, F. Mo, T. Saraya, T. Hiramoto, and M. Kobayashi, “A monolithic 3d integration of rram array with oxide semiconductor fet for in-memory computing in quantized neural network ai applications,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [150] J. Y. Wu, Y. S. Chen, W. S. Khwa, S. M. Yu, T. Y. Wang, J. C. Tseng, Y. D. Chih, and C. H. Diaz, “A 40nm low-power logic compatible phase change memory technology,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [151] X. Xu, Q. Luo, T. Gong, H. Lv, S. Long, Q. Liu, S. S. Chung, J. Li, and M. Liu, “Fully cmos compatible 3d vertical rram with self-aligned self-selective cell enabling sub-5nm scaling,” in _IEEE Symposium on VLSI Technology_ , 2016.
  * [152] X. Xu, L. Tai, T. Gong, J. Yin, P. Huang, J. Yu, D. N. Dong, Q. Luo, J. Liu, Z. Yu, X. Zhu, X. L. Wu, Q. Liu, H. LV, and M. Liu, “40x retention improvement by eliminating resistance relaxation with high temperature forming in 28 nm rram chip,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.
  * [153] C. Xue, T. Huang, J. Liu, T. Chang, H. Kao, J. Wang, T. Liu, S. Wei, S. Huang, W. Wei, Y. Chen, T. Hsu, Y. Chen, Y. Lo, T. Wen, C. Lo, R. Liu, C. Hsieh, K. Tang, and M. Chang, “A 22nm 2mb reram compute-in-memory macro with 121-28tops/w for multibit mac computing for tiny ai edge devices,” in _IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020.
  * [154] T. Yamauchi, Y. Yamaguchi, T. Kono, and H. Hidaka, “Embedded flash technology for automotive applications,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [155] C. F. Yang, C. Y. Wu, M. H. Yang, W. Wang, M. T. Yang, T. C. Chien, V. Fan, S. C. Tsai, Y. H. Lee, W. T. Chu, and A. Hung, “Industrially applicable read disturb model and performance on mega-bit 28nm embedded rram,” in _IEEE Symposium on VLSI Technology_ , 2020.
  * [156] J. Yang, X. Xue, X. Xu, H. Lv, F. Zhang, X. Zeng, M. Chang, and M. Liu, “A 28nm 1.5mb embedded 1t2r rram with 14.8 mb/mm2 using sneaking current suppression and compensation techniques,” in _IEEE Symposium on VLSI Circuits_ , 2020.
  * [157] T. Yang, K. Li, Y. Chiang, W. Lin, H. Lin, and M. Chang, “A 28nm 32kb embedded 2t2mtj stt-mram macro with 1.3ns read-access time for fast and reliable read applications,” in _IEEE International Solid - State Circuits Conference - (ISSCC)_ , 2018.
  * [158] S. Yasuda, K. Ohba, T. Mizuguchi, H. Sei, M. Shimuta, K. Aratani, T. Shiimoto, T. Yamamoto, T. Sone, S. Nonoguchi, J. Okuno, A. Kouchiyama, W. Otsuka, and K. Tsutsui, “A cross point cu-reram with a novel ots selector for storage class memory applications,” in _Symposium on VLSI Technology_ , 2017.
  * [159] H. K. Yoo, J. S. Kim, Z. Zhu, Y. S. Choi, A. Yoon, M. R. MacDonald, X. Lei, T. Y. Lee, D. Lee, S. C. Chae, J. Park, D. Hemker, J. G. Langan, Y. Nishi, and S. J. Hong, “Engineering of ferroelectric switching speed in si doped hfo2 for high-speed 1t-feram application,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2017.
  * [160] S. Yu, Z. Li, P. Chen, H. Wu, B. Gao, D. Wang, W. Wu, and H. Qian, “Binary neural network with 16 mb rram macro chip for classification and online training,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2016.
  * [161] F. Zhang, H. Zhang, P. R. Shrestha, Y. Zhu, K. Maize, S. Krylyuk, A. Shakouri, J. P. Campbell, K. P. Cheung, L. A. Bendersky, A. V. Davydov, and J. Appenzeller, “An ultra-fast multi-level mote2-based rram,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2018.