# MetaSys: A Practical Open-Source Metadata Management System
to Implement and Evaluate Cross-Layer Optimizations

Nandita Vijaykumar* Ataberk Olgun§ Konstantinos Kanellopoulos§ Hasan Hassan§
Mehrshad Lotfi§ Phillip B. Gibbons$\dagger$ Onur Mutlu§

*_University of Toronto_ § _ETH Zürich_ $\dagger$_Carnegie Mellon University_

###### Abstract

This paper introduces the first open-source FPGA-based infrastructure,
MetaSys, with a prototype in a RISC-V system, to enable the rapid
implementation and evaluation of a wide range of cross-layer techniques in
real hardware. Hardware-software cooperative techniques are powerful
approaches to improving the performance, quality of service, and security of
general-purpose processors. They are however typically challenging to rapidly
implement and evaluate in real hardware as they require full-stack changes to
the hardware, system software, and instruction-set architecture (ISA).

MetaSys implements a rich hardware-software interface and lightweight metadata
support that can be used as a common basis to rapidly implement and evaluate
new cross-layer techniques. We demonstrate MetaSys’s versatility and ease-of-
use by implementing and evaluating three cross-layer techniques for: _(i)_
prefetching in graph analytics; _(ii)_ bounds checking in memory unsafe
languages, and _(iii)_ return address protection in stack frames; each
technique requiring only ~100 lines of Chisel code over MetaSys.

Using MetaSys, we perform the first detailed experimental study to quantify
the performance overheads of using a _single_ metadata management system to
enable multiple cross-layer optimizations in CPUs. We identify the key sources
of bottlenecks and system inefficiency of a general metadata management
system. We design MetaSys to minimize these inefficiencies and provide
increased versatility compared to previously-proposed metadata systems. Using
three use cases and a detailed characterization, we demonstrate that a common
metadata management system can be used to efficiently support diverse cross-
layer techniques in CPUs. MetaSys is completely and freely available at
https://github.com/CMU-SAFARI/MetaSys

##  1 Introduction

Hardware-software cooperative techniques offer a powerful approach to
improving the performance and efficiency of general-purpose processors. These
techniques involve communicating key application and semantic information from
the software to the architecture to enable more powerful optimizations and
resource management in hardware. Recent research proposes many such cross-
layer techniques for various purposes, e.g., performance, quality of service
(QoS), memory protection, programmability, security. For example, Whirlpool
[104] identifies and communicates regions of memory that have similar
properties (i.e., data structures) in the program to the hardware, which uses
this information to more intelligently place data in a non-uniform cache
architecture (NUCA) system. RADAR [96] and EvictMe [167] communicate which
cache blocks will no longer be used in the program, such that cache policies
can evict them. These are just a few examples in an increasingly large space
of cross-layer techniques proposed in the form of hints implemented as new ISA
instructions to aid cache replacement, prefetching, memory management, etc.
[66, 126, 58, 26, 167, 22, 137, 179, 136, 116, 117, 96, 159], program
annotations/directives to convey program semantics [3, 104, 47, 87, 58, 163],
or interfaces to communicate an application’s QoS requirements for efficient
partitioning and prioritization of shared hardware resources [93, 62].

While cross-layer approaches have been demonstrated to be highly effective,
such proposals are challenging to evaluate on real hardware as they require
cross-layer changes to the hardware, operating system (OS), application
software, and instruction-set architecture (ISA). Existing open-source
infrastructures for implementing cross-layer techniques in real hardware
include PARD [62, 93] for QoS and Cheri [174] for fine-grained memory
protection and security. Unfortunately, these infrastructures are not designed
to provide key features required for _performance_ optimizations: _(i)_ rich
dynamic hardware-software interfaces, _(ii)_ low-overhead metadata management,
and _(iii)_ interfaces to numerous hardware components such as prefetchers,
caches, etc.

In this work, we introduce MetaSys (Metadata Management System for Cross-Layer
Performance Optimization), a full-system FPGA-based infrastructure, with a
prototype in the RISC-V Rocket Chip system [10], to enable rapid
implementation and evaluation of diverse cross-layer techniques in real
hardware. MetaSys comprises three key components: (1) A rich hardware-software
interface to communicate a _general_ and extensible set of application
information to the hardware architecture at runtime. We refer to this
additional application information as _metadata_. Examples of metadata include
memory access pattern information for prefetching, data reuse information for
cache management, address bounds for hardware bounds checking, etc. The
interface is implemented as new instructions in the RISC-V ISA and is wrapped
with easy-to-use software library abstractions. (2) Metadata management
support in the OS and hardware to store and access the communicated metadata.
Hardware components performing optimizations can then efficiently query for
the metadata. We use a _tagged memory-based_ design for metadata management
where each memory address is tagged with an ID. This ID points to metadata
that describes the data contained in the location specified by the memory
address. (3) Modularized components to quickly implement various cross-layer
optimizations with interfaces to the metadata management support, OS, core,
and memory system. Our FPGA-based infrastructure provides flexible modules
that can be easily extended to implement different cross-layer optimizations.

The closest work to our proposed system is XMem [164]. XMem proposes a general
metadata management system that can communicate semantic information at
compile time. This limits the use cases supported by XMem. MetaSys has the
following benefits over XMem: First, MetaSys offers a richer interface that
communicates a flexible amount of metadata at _runtime_ , rather than being
limited to statically available program information. This enables a wider set
of use cases and more powerful cross-layer techniques (as explained in §3.8).
Second, MetaSys has a more optimized system design that is designed to be
_lightweight_ in terms of the hardware complexity and changes to the ISA,
without sacrificing versatility (§3.8). MetaSys incurs only a small area
overhead of 0.02% (including 17KB of additional SRAM), 0.2% memory overhead in
DRAM, and adds only 8 new instructions to the RISC-V ISA. Third, MetaSys is
open-source and freely available, whereas XMem is neither implemented nor
evaluated in real hardware with full-system support.

Use cases. Cross-layer techniques that can be implemented with MetaSys include
performance optimizations such as cache management, prefetching, memory
scheduling, data compression, and data placement; cross-layer techniques for
QoS; and lightweight techniques for memory protection (see §7). To demonstrate
the versatility and ease-of-use of MetaSys in implementing new cross-layer
techniques, we implement and evaluate three hardware-software cooperative
techniques: _(i)_ prefetching for graph analytics applications; _(ii)_ bounds
checking in memory unsafe languages, and _(iii)_ return address protection in
stack frames. These techniques were quick to implement with MetaSys, each
requiring only an additional ~100 lines of Chisel [13] code on top of
MetaSys’s hardware codebase (~1800 lines of code).

Characterizing a general metadata management system. Using MetaSys, we perform
the first detailed experimental characterization and limit study of the
performance overheads of using a _single_ common metadata management system to
enable multiple diverse cross-layer techniques in a general-purpose processor.
We make four new observations from our characterization across 24 applications
and 4 microbenchmarks that were designed to stress MetaSys.

First, the performance overheads from the cross-layer interface and metadata
system itself are on average very low (2.7% on average, up to 27% for the most
intensive microbenchmark). Second, there is no performance loss from
supporting _multiple_ techniques that simultaneously query the shared metadata
system. This indicates that MetaSys can be designed to be a scalable
substrate. Third, the most critical factor in determining the performance
overhead is the fundamental spatial and temporal locality in the accesses to
the metadata itself. This determines the effectiveness of the metadata caches
and the additional memory accesses to retrieve metadata. Fourth, we identify
TLB misses from the required address translation when metadata is retrieved
from memory as an important factor in performance overhead.

Conclusions from characterization. From our detailed characterization and
implemented use cases on real hardware, we make the following conclusions:
First, using a _single_ general metadata management system is a promising low-
overhead approach to implement _multiple_ cross-layer techniques in future
general-purpose processors. The significance of using a single framework is in
enabling a wide range of cross-layer techniques with a single change to the
hardware-software interface [164, 93] and _consolidating_ common metadata
management support; thus, making the adoption of new cross-layer techniques in
future processors significantly easier. Second, we demonstrate that a common
framework can simultaneously and scalably support multiple cross-layer
optimizations. For our implemented use cases, we observe low performance
overheads from using the general MetaSys system: 0.2% for prefetching, 14% for
bounds checking, and 1.2% for return address protection.

MetaSys is fully open-source and freely available at
https://www.github.com/CMU-SAFARI/MetaSys.

This work makes the following major contributions.

  * •

We introduce MetaSys, the first full-system open-source FPGA-based
infrastructure of a lightweight metadata management system. MetaSys provides a
rich hardware-software interface that can be used to implement a diverse set
of cross-layer techniques. We implement a prototype of MetaSys in a RISC-V
system providing the required support in the hardware, OS, and the ISA to
enable quick implementation and evaluation of new hardware-software
cooperative techniques in real hardware.

  * •

We propose a new hardware-software interface that enables _dynamically_
communicating information and a more streamlined system design that can
support a richer set of cross-layer optimizations than prior work [164].

  * •

We present the first detailed experimental characterization of the performance
and area overheads of a _general_ hardware-software interface and lightweight
metadata management system designed to enable _multiple_ and diverse cross-
layer performance optimizations. We identify key sources of inefficiencies and
bottlenecks of a general metadata system on real hardware, and we demonstrate
its effectiveness as a common substrate for enabling cross-layer techniques in
CPUs.

  * •

We demonstrate the versatility and ease-of-use of the MetaSys infrastructure
by implementing and evaluating three hardware-software cooperative techniques:
_(i)_ prefetching for graph analytics applications; _(ii)_ efficient bounds
checking for memory-unsafe languages; and _(iii)_ return address protection
for stack frames. We highlight other use cases that can be implemented with
MetaSys.

##  2 Background and Related Work

Hardware-software cooperative techniques in CPUs. Cross-layer performance
optimizations communicate _additional_ information across the application-
system boundary. We refer to this information as _metadata_. Metadata that is
typically useful for performance optimization include program properties such
as access patterns, read-write characteristics, data locality/reuse, data
types/layouts, data "hotness", and working set size. This metadata enables
more intelligent hardware/system optimizations such as cache management, data
placement, thread scheduling, memory scheduling, data compression, and
approximation [164, 163, 162]. For QoS optimizations, metadata includes
application priorities and prioritization rules for allocation of resources
such as memory bandwidth and cache space [93, 62, 107, 106, 76, 48, 153, 152].
Memory safety optimizations may communicate base/bounds addresses of data
structures [45, 43].

A _general_ framework is a promising approach as it enables many cross-layer
techniques with a single change to the hardware-software interface and enables
_reusing_ the metadata management support across multiple optimizations. Such
systems were recently proposed for performance [164, 163], memory protection
and security [45, 174], and QoS [93, 62].

A general framework to support a wide range of cross-layer
optimizations—specifically for _performance_ —requires: _(i)_ a rich and
dynamic hardware-software interface to communicate a diverse set of metadata
at runtime and _(ii)_ lightweight and _low-overhead_ metadata management
[164], and _(iii)_ interfaces to numerous hardware components. Even small
overheads imposed as a result of the system’s generality may overshadow the
performance benefits of a cross-layer technique. General metadata systems may
also impose significant complexity, performance, and power overheads to the
processor. While prior work has demonstrated the significant benefits of
cross-layer approaches, no previous work has characterized the efficiency and
capacity limits of a general metadata system for cross-layer optimizations in
CPUs.

Tagged architectures. MetaSys is inspired by the metadata management and
interfaces proposed in XMem [164] and the large body of work on tagged memory
[173, 68, 45, 53, 182] and capability-based systems [174, 27, 85, 168]. We
compare against the closest prior work, XMem, qualitatively in §3.8 and
quantitatively in §5. Unlike all above works, our goal is to provide an open-
source framework to implement and these prior cross-layer approaches in real
hardware and to perform a detailed real-system characterization of such
metadata systems for performance optimization.

Infrastructure for evaluating cross-layer techniques. Evaluating the overheads
and feasibility of a newly-proposed cross-layer technique is non-trivial.
Fully characterizing the performance and area overheads either with a full-
system cycle-accurate simulator or an FPGA implementation requires
implementing: _(i)_  _Hardware support_ to implement the mechanism; _(ii)_
_OS support_ for OS-based cross-layer optimizations and to characterize the
context-switch and system overheads of saving and handling a process’
metadata; and _(iii)_  _Compiler support and ISA modifications_ to add and
recognize new instructions to communicate metadata.

Recent works propose general systems that are designed to enable cross-layer
techniques for QoS (PARD [62, 93]) or fine-grained memory protection and
security (Cheri [174]). PARD enables tagging of components and applications
with IDs that are propagated with memory requests and enforcing QoS
requirements in hardware. Cheri [174] is a capability-based system that
provides hardware support and ISA extensions to enable fine-grained memory
protection. Neither system supports the _(i)_ communication of diverse
metadata at runtime, _(ii)_ flexible granularity tagging of memory to enable
efficient metadata lookups from multiple components, or _(iii)_ interfaces to
numerous hardware components (such as the prefetcher, caches, memory
controllers) that are needed for _performance_ optimization.

Our Goal. Our goal in this work is twofold. First, we aim to develop an
efficient and flexible _open-source_ framework that enables rapid
implementation of new cross-layer techniques to evaluate the associated
performance, area, and power overheads, and thus their benefits and
feasibility, in real hardware.

Second, we aim to perform the first detailed limit study to characterize and
experimentally quantify the overheads associated with _general_ metadata
systems to determine their practicality for performance optimization in future
CPUs.

##  3 MetaSys: Enabling and Evaluating Cross-layer Optimizations

To this end, we develop MetaSys (Metadata Management System for Cross-Layer
Performance Optimization), an open-source full-system FPGA-based
infrastructure to implement and evaluate new cross-layer techniques in real
hardware. MetaSys includes: _(i)_ a rich hardware-software interface to
dynamically communicate a flexible amount of metadata at runtime from the
application to the hardware, using new RISC-V instructions; _(ii)_ a tagged
memory-based [173, 68, 45, 53, 182] implementation of metadata management in
the system and OS; and _(iii)_ flexible modules to add new hardware
optimizations with interfaces to the metadata, processor, memory, and OS. We
build a prototype of MetaSys in the RISC-V Rocket Chip [10] system.

We choose an FGPA implementation as opposed to a full-system simulator as:
_(i)_ This enables us to focus on feasibility as all components need to be
fully implemented (e.g., ports, wires, buffers) and their impact on area,
cycle time, power, and scalability is quickly visible. _(ii)_ FPGAs are much
faster, running full application simulations in a few minutes/hours as opposed
to many days on a full-system simulator, making FPGAs a better fit for quick
experimentation. _(iii)_ The RTL generated can be used for more accurate area
and power calculation and potential future synthesis on other systems.

Fig. 1 depicts an overview of the major hardware components in MetaSys and
their operation: The mapping management unit ❶, the optimization client ❷, and
the metadata lookup unit ❸.

Figure 1: MetaSys hardware components and operation. MetaSys’ structures are
highlighted.

###  3.1 Tagged Memory-based Metadata Management

Similar to prior systems for taint-tracking, security, and performance
optimization, MetaSys implements _tagged memory_ -based [173, 68, 53, 182]
metadata management. MetaSys associates metadata with memory address ranges of
arbitrary sizes by tagging each memory address with an 8-bit (configurable) ID
or tag. Each tag is a unique pointer to metadata that describes the data at
the memory address. Hardware optimizations (e.g., in the cache, memory
controller, or core) can query for the tag associated with any memory address
and the metadata associated with the tag.

The mapping between each memory address and the corresponding ID is saved in a
table in memory referred to as Metadata Mapping Table (MMT): ❹ in Fig. 1. This
table is allocated by the OS for each process and is saved in memory. In
MetaSys (similar to XMem [164] and Cheri [174]), we tag _physical addresses_.
As a result, any virtual address has to be translated before indexing the MMT
to retrieve the tag ID. To enable fast retrieval of IDs, we implement a cache
for the MMT in hardware that stores frequently accessed mappings, referred to
as the Metadata Mapping Cache (MMC) ❺. MMC misses lead to memory accesses to
retrieve mappings from the MMT in memory.

MetaSys can be configured to tag memory at flexible granularities. In §9.1, we
evaluate the performance impact of the tagging granularity. The size of the
MMT depends on the tagging granularity. For a 512B mapping granularity, the
MMT requires 0.2% of physical memory (16MB in a 8GB system). The MMC holds 128
entries, where each entry stores a physical-address-to-tag mapping, and is
608B in size (8 bit entry and 30 bit tag).

We implement dedicated mapping tables for tag IDs rather than use the page
table or TLBs for the following reasons: First, doing so obviates the need to
modify the latency-critical address translation structures. Second, MetaSys
associates physical addresses with Tag IDs rather than virtual addresses (to
enable the memory controller and LLCs to look up metadata). Thus, a page table
or TLB cannot be directly used to save Tag IDs as they are indexed with
virtual addresses.

The actual metadata associated with any ID is saved in special SRAM caches
that are private to each hardware component or optimization. For example, the
prefetcher would separately save access pattern information, while a hardware
bounds checker would privately save data structure boundary information. We
refer to these stores as Private Metadata Tables (PMTs) ❻. The PMTs are saved
near each component (private to each component) and are loaded/updated by
MetaSys. The metadata (e.g., locality/”hotness”) is encoded such that it can
be directly interpreted by the component, e.g., a prefetcher.

###  3.2 The Hardware-Software Interface

Communicating application information with MetaSys requires _(i)_ associating
memory address ranges with a tag or ID of configurable size (8 bits by
default) and _(ii)_ associating each ID with the relevant metadata. The
metadata could include program properties that describe the memory range, such
as data locality/reuse, access patterns, read-write characteristics, data
"hotness", and data types/layouts. We use two operators (described below) that
can be called in programs to dynamically communicate metadata.

To associate memory address ranges with an ID, we provide the MAP/UNMAP
interface ❼ (similar to XMem [164]). MAP and UNMAP are implemented as new
RISC-V instructions that are interpreted by the Mapping Management Unit (MMU)
to map a range of memory addresses (from a given virtual address up to a
certain length) to the provided ID. These mappings are saved by the MMU in the
MMT. We also implement 2D and 3D versions of MAP to efficiently map
2/3-dimensional address ranges in a multi-dimensional data structure with a
single instruction.

To associate each ID with metadata, we provide the CREATE interface. CREATE ❽
takes 3 inputs from the application: the tag ID, the 8-bit ID for the
_hardware component_ (i.e., prefetcher, bounds checker, etc., called Module
ID), and 512B of metadata. CREATE directly populates the PMT of the
appropriate hardware component with 512 bytes of (or less) metadata. Each PMT
(private to the optimization client) has 256 entries assuming 8-bit tag IDs.
The CREATE operator overwrites the metadata at the entry indexed by the tag ID
at the PMT specified by the module ID. All CREATE and MAP instructions are
associated with the _next_ load/store instruction in program order to avoid
inaccuracies due to out-of-order execution. In other words, an implicit
dependence is created in hardware between these instructions and the next
load/store, and they are committed together. This enables associating
information with the next load/store and not just the memory region associated
with it, e.g., in the bounds checking use case described in §6.1.

Table LABEL:table:semantics lists the new instructions along with their
arguments.

Table 1: MetaSys instructions. MetaSys Operator |  MetaSys ISA Instructions
---|---
CREATE |  CREATEClientID, TagID, Metadata
(UN)MAP |  (UN)MAP TagID, start_addr, size
(UN)MAP2D TagID, start_addr, lenX, sizeX, sizeY
(UN)MAP3D TagID, start_addr, lenX, lenY, sizeX, sizeY, sizeZ ;

###  3.3 Metadata Lookup

Each optimization component is triggered by a hardware event ❾ (e.g., a cache
miss). A component then retrieves the physical address corresponding to the
virtual address associated with the event (e.g., the virtual address that
misses in the cache) from the TLB ❿ (in case of L1 optimizations) and queries
the MMC with the physical address to retrieve the associated tag ID. On a miss
in the MMC, the mapping is retrieved from the MMT in memory. The optimization
client uses the retrieved tag ID to obtain the appropriate metadata from the
PMT. The optimization client is designed to flexibly implement a wide range of
use cases and can be designed based on the optimization at hand. For example,
the optimization client used to build the prefetcher use case in §5 has
interfaces to the prefetcher, caches, memory controller, and TLBs to make
implementing optimizations easier. Each client has a static ID (clientID) and
a PMT that is updated by the CREATE operator.

###  3.4 Operating System Support

We add OS support for metadata management in the RISC-V proxy kernel [128],
which can be booted on our Rocket RISC-V prototype: First, we add support to
manage the MMT in memory, where the OS allocates the MMT in the physical
address space and communicates the pointer to the MAP hardware support.
Second, we add support to flush the PMTs during a context switch (similar to
how the TLB is flushed). Third, if the OS changes the virtual to physical
address mapping of a page, then to ensure consistency of the metadata, the MMT
is updated by the OS to reflect the correct physical-address-to-tag-ID mapping
and the corresponding MMC entries are invalidated. We modify the page
allocation mechanism in the OS to do this. In addition, we also provide
support to implement optimizations performed by the OS or with OS cooperation.
To do so, MetaSys enables trapping into the OS to perform customized checks or
optimizations (e.g., protection checks or altering virtual-to-physical
mappings) based on specific hardware trigger events (using interrupt
routines). We describe one such use case in §6.

###  3.5 Coherence/Consistency of Metadata in Multicore Systems

MetaSys can be flexibly extended to multicore processors. Metadata is
maintained at a process-level, therefore, threads within the same process
cannot have different metadata for the same data structure. The MMC is a per-
core structure, while the Private Metadata Tables (PMTs) are per-component
structures (e.g, at the memory controller, LLC, prefetcher). The two dynamic
operators (CREATE and MAP) may cause challenges in coherence and consistency
of metadata in multicore systems. CREATE directly updates metadata associated
with the _per-process_ tag ID, which is saved at the per-component PMTs. The
PMTs are shared by all cores when the optimization component is also shared
(and thus any updates by CREATE are automatically coherent). The PMTs for
private components (e.g., L1 cache) are not coherent and can only be updated
by the corresponding thread. MAP updates the mapping in the MMC, which is
private to each core. To ensure coherence of the MMC mappings, a MAP update
invalidates the corresponding MMC entry (if present) in other MMCs by
broadcasting updates with a snoopy protocol. If the use case requires
consistency of the metadata, i.e., ordering between a CREATE/MAP instruction
and when it is visible to other cores, barriers and fence instructions are
used to enforce any required ordering between threads for updates to metadata.

###  3.6 Timing Sensitivity of Metadata

MetaSys supports three modes: _(i)_  _Force stall_ , where the instruction
triggering a metadata lookup cannot commit until the optimization completes
(e.g., for security use cases); _(ii)_  _No stall_ , where metadata lookups do
not stall the core but are always resolved (e.g., for page placement, cache
replacement), and _(iii)_  _Best effort_ , where lookups may be dropped to
minimize performance overheads (e.g., for prefetcher training).

###  3.7 Software Library

We develop a software library that can be included in user programs to
facilitate the use of MetaSys primitives CREATE and MAP (Table 2). The library
exposes three functions: _(i)_ CREATE populates an entry indexed by the tag ID
(TagID) in the PMT of a hardware optimization client (ClientID) with the
corresponding metadata; _(ii)_ MAP updates the MMT by assigning tag IDs to
memory addresses of the range (start, end); _(iii)_ UNMAP resets the tag IDs
of the corresponding address range in the MMT. While the operators can be
directly used via the provided software library, their use can be simplified
by using wrapper libraries that abstract away the need to directly manage tag
IDs and their mappings.

Table 2: The MetaSys software library function calls. Library Function Call | Description
---|---
CREATE(ClientID, TagID, *meta) |  ClientID -> PMT[TagID] = *metadata
MAP(start*, end*, TagID) |  MMT[start…end] = TagID
UNMAP(start*, end*) |  MMT[start…end] = 0

###  3.8 Comparison to the XMem Framework [164]

MetaSys implements a tagged-memory-based system with a metadata cache similar
to XMem [164]. MetaSys however has three major benefits over XMem. First,
MetaSys enables communicating metadata at _runtime_ using a more powerful
CREATE operator that is implemented as a new instruction. In XMem, metadata is
communicated only _statically_ at compile time (CREATE is hence a compiler
pragma). MetaSys thus enables a wider set of optimizations including fine-
grained memory safety, protection, prefetching, etc., and enables
communicating metadata that is dependent on program input and metadata that
can be accurately known only at runtime (e.g., access patterns, data
"hotness", etc.). MetaSys was designed to efficiently handle these dynamic
metadata updates. Second, the dynamic and more expressive CREATE operator
obviates the need for additional interfaces (ACTIVATE/DEACTIVATE) to track the
validity of statically communicated metadata. This enables a more streamlined
metadata system in MetaSys with fewer new instructions, tables, and lookups.
Third, MetaSys allows the application programmer to directly select which
cross-layer optimization to enable/disable and communicate metadata to, via
the CREATE operator. XMem, on the other hand, does not allow control of
hardware optimizations from the application. Table 3 summarizes the MetaSys
operators and compares to the corresponding operators in XMem. _Of the three
MetaSys use cases we evaluate in this paper, only return address protection (§
6.2) can be implemented with XMem._

Table 3: Comparison between MetaSys and XMem interfaces. Operator |  XMem [164] |  MetaSys
---|---|---
CREATE |  Compiler pragma to communicate static metadata at program load time. |  Selects a hardware optimization, dynamically associates metadata with an ID, and communicates both to hardware at runtime (implemented as a new instruction).
(UN)MAP |  Associate memory ranges with tag IDs (implemented as new instructions). |  Same semantics and implementation as XMem.
(DE)ACTIVATE |  Enable/disable optimizations associated with a tag ID (implemented as new instructions). |  Does not exist as the same functionality can now be done with CREATE.

###  3.9 FPGA-based Infrastructure

We build a full system prototype of MetaSys on an FPGA with the Rocket Chip
RISC-V system [10] and add the necessary support in the compiler, libraries,
OS, ISA, and hardware. The modularized MetaSys components can also be ported
to other RISC-V cores. We used the RoCC accelerator [10] in the Rocket chip to
implement the metadata management system. RoCC is a customizable module that
enables interfacing with the core and memory. The hardware support implemented
in ROCC comprises _(i)_ the control logic to handle MAPs and CREATEs, _(ii)_
control logic to perform metadata lookups by components that implement
optimizations, and _(iii)_ the memory for metadata caches (MMC and PMTs). We
extended the RISC-V ISA with 8 instructions (6 for MAP/CREATE and 2 for OS
operations). To implement all the hardware modules of MetaSys, we
modified/added 1781 lines of Chisel code in the Rocket Chip. As we demonstrate
later, since the MetaSys hardware modules can be flexibly reused across
multiple hardware-software optimizations, the techniques in our use cases only
required 87-103 additional lines of Chisel code. The full MetaSys
infrastructure is open-sourced [57] including the Chisel code for the MetaSys
hardware support, the RISC-V OS with the required modification, and the
software libraries to expose the MetaSys primitives.

###  3.10 Implementing a Hardware-Software Cooperative Technique with MetaSys

To implement a new hardware technique with the baseline MetaSys code, we
provide a flexible module (❶ in Fig. 2) with a PMT and interfaces to the
metadata lookup unit, to the core (to receive triggers), and interfaces to the
cache controller. The interface to the lookup unit ❷ provides dynamic access
to the metadata communicated by the CREATE and MAP operators. The interfaces
to the core ❸ and the memory system ❹ can be used as _trigger_ events for
optimization and lookups (e.g., a cache miss). The different components within
the MetaSys logic itself (i.e., the metadata caches, logic to access the
Metadata Mapping Table in memory, and the lookup logic) can be flexibly
reconfigured.

###  3.11 Dynamically-Typed or Managed Languages

MetaSys relies heavily on function calls/libraries that abstract away low
level details that call the MetaSys instructions even in C/C++. With managed
and dynamically-typed languages, the metadata associated with data
structures/objects would be provided by the user with additional class/object
member functions. The metadata could also be directly embedded within
object/class definitions (e.g., a list or map in Python would by definition
have certain access properties). Other properties (e.g., data types) would be
provided by the interpreter (in the case of dynamically-typed languages) and
the mapping/remapping calls to memory addresses would be handled by the
runtime during memory (de)allocation.

Figure 2: MetaSys Optimization Client.

###  3.12 Comparison to Specialized Cross-Layer Solutions

In comparison to specialized cross-layer solutions, MetaSys offers the
following benefits: (i) Generality: towards implementing a large number of use
cases, including more complex use cases such as specialized prefetching (§5),
which amortizes the overall hardware cost; (ii) Flexibility and versatility in
the implemented instructions: A challenge with specialized cross-layer
solutions is the need to add new instructions that create challenges in
forward/backward compatibility and also require changes across the stack for
each new optimization. With MetaSys, the instructions are designed to be
agnostic to the optimization and only require a one-off change to the
hardware-software interface; (iii) Infrastructure for evaluation: MetaSys can
be used to implement many specialized cross-layer techniques in real hardware,
which would otherwise be a challenging programming task (as demonstrated in §5
and §6). In §5 and §6, we evaluate MetaSys’s ability to implement several
cross-layer techniques.

##  4 Methodology

Baseline system. We use the in-order Rocket core [10] as the baseline CPU and
conduct our experiments on the ZedBoard Zynq-7000 [12] FPGA board. Table 4
lists the parameters of the core and memory system as well as evaluated
workloads.111Since DRAM is disproportionately faster than the CPU clock rate
on FPGAs, we added logic in the memory controller to scale the rate at which
memory requests are issued. The resulting average memory latency and bandwidth
in core cycles were validated with microbenchmarks against a real CPU. MetaSys
does not require any changes to support an L2/LLC and optimization modules for
an L2/LLC can be flexibly implemented similar to the L1. The cost of an MMC
miss may be further alleviated with an L2/LLC that reduces access to memory.

Table 4: Parameters of the evaluated real FPGA-based system.

CPU: 25 MHz; in-order Rocket core [10]; TLB 16 entries DTLB; LRU policy;
---
L1 Data + Inst. Cache: 16 KB, 4-way; 4-cycle; 64 B line; LRU policy; MSHR
size: 2
MMC: NMRU Policy; 128 entries; 38bits/entry; Tagging Granularity: 512B;
Private Metadata Table: 256 entries; 64B/entry; DRAM: 533MHz; $V_{dd}$: 1.5V;
Workloads: Ligra [145]: PageRank (PR), Shortest Path (SSSP), Collaborative
Filtering (CF)
Teenage Follower (TF), Triangle Counting (TC), Breadth-First Search (BFS),
Radius Estimation (Radii)
Connected Components (CC); Polybench [124]; $\mu$Benchmarks

##  5 Use Case 1: HW-SW Cooperative Prefetching

Hardware-software cooperative prefetching techniques have been widely proposed
to handle challenging access patterns such as in graph processing [18, 185,
181, 103, 113, 5, 6, 157, 4], pointer-chasing [131, 7, 30, 187, 49, 132],
linear algebra computation [32] and other applications [121, 9, 166, 165]. In
this section, we demonstrate how MetaSys can be flexibly used to implement and
evaluate such prefetching techniques. We design a new prefetcher for graph
applications that leverages knowledge of the semantics of graph data
structures using MetaSys. Graph applications typically involve irregular
pointer-chasing-based memory access patterns. The data-dependent non-
sequential accesses in these workloads are challenging for spatial [55, 14,
54, 71, 64, 150, 82, 148, 52, 75, 140, 125, 99, 80, 21, 139, 23, 115],
temporal [70, 169, 147, 170, 15, 65, 176, 175, 171, 31, 33, 51, 61, 19, 35,
73] and learning-based hardware prefetchers [20, 122, 120, 59, 141, 143, 142,
184] that rely either on (i) program context information (e.g. program
counter, cache line address) or (ii) memorizing long sequences of cache line
addressses to generate accurate prefetch requests.

_To implement the hardware support for our prefetcher, we only needed to add
87 lines of Chisel code to the baseline MetaSys codebase, all within the
provided module for new optimization components._

Vertex-centric graph analytics typically involves first traversing a _work
list_ containing vertices to be visited (❶ in Fig. 3). For each vertex, the
application accesses the _vertex list_ ❷ to retrieve the neighboring vertex
IDs from the _edge list_ ❸. To perform computation on the graph, the
application then operates on the properties of these neighboring vertices
(retrieved from the property list ❹). Graph processing thus involves a series
of memory accesses that depend on the contents of the work, vertex and edge
lists.

Figure 3: Data-dependent accesses in vertex-centric graph processing model.

In this use case, we design a prefetcher that can interpret the contents of
each of the above data structures and appropriately compute the next data-
dependent memory address to prefetch. To capture the required application
information for each data structure, we use MetaSys’s CREATE interface to
communicate the following metadata: _(i)_ base address of the data structure
that is indexed using the current data structure’s contents (64 bits); _(ii)_
base address of the current data structure (64 bits); _(iii)_ data type (32
bits) and size (32 bits) to determine the index of the next access; and _(iv)_
the prefetching stride (6 bits). MAP then associates the address range of each
data structure with the appropriate tag.

Listing 1 shows a detailed end-to-end example of how metadata is created in
the application (BFS), how metadata tags are associated with the data
structures of BFS, and how the prefetcher operates. Lines 3-12 (incorporated
into the code of the BFS application) use the MetaSys software libraries to
create metadata (with CREATE) and associate it with the corresponding data
structures (using MAP). CREATE saves the metadata in the PMT and MAP updates
the MMT. Lines 16-34 (incorporated into the hardware optimization client
responsible for prefetching) describe the algorithm behind the MetaSys-based
prefetcher. The prefetcher is implemented with an optimization client
(ClientID = 0). The prefetcher essentially: _(i)_ snoops every memory request
from the core and retrieves the associated tag ID using MetaSys; _(ii)_
queries the PMT to retrieve the communicated metadata (listed above); and
_(iii)_ uses the metadata to identify dependencies between the data structures
of the application.

We describe a detailed walkthrough of how the prefetcher operates during the
execution of the BFS application using Fig. 3 and Listing 1. In Fig. 3, when
the prefetcher snoops a memory request that targets the work list at index 0,
it looks ahead (depending on the prefetching stride) to retrieve the contents
of the work list at index 1. At this point, it also prefetches the contents of
the vertex, edge, and property lists based on the computed index at each
level. In graph applications where the work list is ordered, the prefetcher is
configured to simply stream through the contents of the vertex and edge lists
to prefetch the data dependent memory locations in the property list. The
$snoop\\_mem\\_request(address)$ (Line 16) function is executed for each
request sent by the core to the memory hierarchy. For every memory request,
the prefetcher accesses the MMC using the address to receive the tag ID (using
MetaSys’s lookup functionality). Next, it indexes the PMT using the tag ID to
retrieve the metadata associated with the memory request. Using the metadata,
the prefetcher determines if the request comes from one of the data structures
of the application (Line 24). In this case, the prefetcher first prefetches
ahead (Line 26) according to the stride and waits until it receives the value
of the prefetched request (Line 28). Using the value, it calculates the
address of the data-dependent data structure (e.g., value of WorkList used as
an index for VertexList) and looks up the metadata for the newly-formed
address. The same procedure happens until no further data-dependency is found
(Line 22).

The prefetcher can be flexibly configured (by associating metadata to data
structures, Lines 3-12 in Listing 1) by the user based on the specific
properties associated with any data structure, algorithm, and the desired
aggressiveness of prefetcher.

⬇

1/* Additional Code in BFS */

2// Create metadata (arguments: ClientID=0, tag ID, metadata)

3metadata_create(0, 1, WorkList, { sizeof(Worklist), VertexList, Stride });

4metadata_create(0, 2,VertexList, { sizeof(VertexList), EdgeList, Stride});

5metadata_create(0, 3,EdgeList, { sizeof(Worklist), VertexList, Stride});

6metadata_create(0, 4,Property, {sizeof(Property), NULL, Stride});

7

8// Map tag 1 to Worklist

9metadata_map((void*) (WorkList), mapSize, 1);

10metadata_map((void*) (VertexList), mapSize, 2);

11metadata_map((void*) (EdgeList), mapSize, 3);

12metadata_map((void*) (Property), mapSize, 4);

13

14/* Hardware Prefetcher Functionality */

15// snoop every memory request

16void snoop_mem_request ( address ):

17 // Access MetaSys using the address

18 (Valid,Base,Bounds,

19 PointerToNextDS,Stride) = metadata_lookup (address);

20

21 // While the data structure traversal is not complete

22 while (Valid && PointerToNext != NULL)

23 // If the memory request comes from a tracked data structure

24 if (Base < address && address > Bounds)

25 // Initiate a stride prefetch request

26 initiate_prefetch(address+stride);

27 // Wait for the prefetch request to return data

28 Value = wait_for_value(address+stride);

29 // Discover the address of the next data structure (DS)

30 address = &PointerToNextDS[value];

31 // Look up metadata for the next data structure (DS)

32 nextAddress = PointerToNextDS[value];

33 (Valid,Base,Bounds,

34 PointerToNextDS,Stride) = metadata_lookup (nextAddress);

Listing 1: Metasys-based Graph Prefetcher. Available online [57].

We evaluate the MetaSys-based prefetcher using 8 graph analytics workloads
from the Ligra framework [145] using the Rocket Chip prototype of MetaSys with
the system parameters listed in Table 4. We evaluate three configurations:
_(i)_ the baseline system with a hardware stride prefetcher [55]; _(ii)_
_GraphPref_ , a customized hardware prefetcher that implements the same idea
described above without the generalized MetaSys support (similar to prior work
[5, 157]); and _(iii)_ the MetaSys-based graph prefetcher. In the case of
_GraphPref_ , all the required metadata (e.g., base and bound addresses,
stride) are directly provided to the prefetcher using specialized
instructions. Thus, _GraphPref_ is able to access metadata at low latency and
does not access the memory hierarchy. The prefetcher works in the same way as
the MetaSys-based prefetcher, however, in the case of MetaSys, the general
CREATE/MAP instructions are used to communicate information and the metadata
lookups access the MMC (which may lead to additional memory accesses when
there is an MMC miss). Fig. 4 depicts the corresponding speedups, normalized
to the baseline. We observe that the MetaSys graph prefetcher improves
performance by 11.2% on average (up to 14.3%) over the baseline by accurately
prefetching data-dependent memory accesses. It also significantly outperforms
the stride prefetcher which is unable to capture the irregular access patterns
in graph workloads. Compared to _GraphPref_ , the MetaSys-based prefetcher
performs almost as well: within 0.2% on average (within 0.8% for BFS). The
additional overheads of MetaSys come from the MMC misses and the larger number
of instructions used. In terms of area, MetaSys requires 17KB of SRAM (1KB for
the MMC and 16KB for the Private Metadata Table) compared to the custom
hardware prefetcher which requires 8KB of SRAM for the metadata. The custom
prefetcher requires 2 additional instructions and additional logic to perform
metadata lookups and create/update metadata. We found the area complexity to
be slightly less for the custom solution as the SRAM requirements are lower
($\sim$0.01% for custom hardware versus $\sim$0.02% for MetaSys, compared to a
22nm Intel CPU Core [144]). However, MetaSys’s overhead can be amortized over
multiple use cases, whereas a custom solution is specific to a single use
case.

Figure 4: Speedup with the MetaSys prefetcher.

We conclude that MetaSys can be used to flexibly implement and evaluate
hardware-software cooperative techniques for prefetching by leveraging
MetaSys’s metadata support and interfaces, incurring only small overheads from
MetaSys’s general metadata management.

##  6 Use Case 2: Memory Safety and Protection

We describe two hardware-software cooperative mechanisms for memory safety and
protection that can be directly implemented with MetaSys. _To implement both
use cases, we only add 103 lines of Chisel code to the baseline MetaSys code,
all within the new optimization client that is used in both use cases._

###  6.1 Hardware Bounds Checking

Unmanaged languages such as C/C++ provide great flexibility in memory
management but an important challenge with these languages is _memory safety_.
The pointer casting and pointer arithmetic supported by these languages allow
buffer overflows and potentially hazardous writes to arbitrary memory
locations. Prior work has demonstrated a range of software approaches [177,
180, 112, 11, 34, 46, 67, 119, 111, 60, 133, 100, 114, 50, 173, 79] to
increase memory safety in the form of static or dynamic checks, such as CCured
[112], Cyclone [67], and Softbound [111]. These approaches are known to incur
significant runtime overheads in performing numerous checks in software [156].
Hardware-based approaches offer a promising opportunity to alleviate these
overheads. Prior work [43, 84, 110, 109, 174, 42, 161, 98], including
HardBound [43], ShaktiT [98], and Cheri [174] investigate enabling hardware-
software cooperative bounds checking. These approaches require architectures
that are entirely specialized for bounds checking [43, 84, 98] or more
heavyweight metadata management systems tailored for memory security and
protection [174, 161, 109, 110, 42]. In this section, we demonstrate how
MetaSys can be used to implement hardware-based bounds checking at low
overhead using a lightweight and general metadata system.

####  6.1.1 Implementing bounds checking with MetaSys.

We use Listing 2, where two arrays A and B are traversed with a stride of one
element, to illustrate the mechanism. To implement bounds checking with
MetaSys, we use the MAP operator to tag each data structure to be protected
with a unique ID (lines 3-5). For dynamically allocated nodes (which may not
be contiguously located), each node is tagged with the _same_ ID as other
nodes in the same data structure. Every memory access in the program then
needs to be verified in hardware to be going to the correct data structure. To
do this, we add the CREATE operator before every load or store to a protected
data structure (lines 11, 14, and 18). The CREATE operator in this case
communicates the tag ID of the desired data structure as metadata and the
ClientID of the bounds checking hardware. In hardware, we simply check whether
there is a match between CREATE’s tag ID and the ID of the load/store address
that follows the CREATE instruction. To perform this check, the bounds check
optimization client (ClientID=0), performs a lookup to the MMC to retrieve the
tag ID associated with the load/store address. This ID is compared with the
value stored in the PMT by the previous CREATE instruction. If there is a
mismatch, this indicates a buffer overflow or an access to data that is not
part of the intended data structure as the load is accessing data that was not
mapped to the same tag ID and using its interface to the OS, MetaSys
terminates the program.

⬇

1/* Example bounds checking software */

2// Map TagIDs to three arrays

3metadata_map((void*) (array1), mapSize, 1);

4metadata_map((void*) (array2), mapSize, 2);

5metadata_map((void*) (array3), mapSize, 3);

6

7// Access every element of each array with a stride of one element

8for(i = 0 ; i < array_size ; i += 1)

9{

10 // Create metadata with TagID=1 and Metadata=1 to ClientID=0

11 metadata_create(0,1,1);

12 int elem1 = array1[i];

13 // Create metadata with TagID=2 and Metadata=2 to ClientID=0

14 metadata_create(0,2,2);

15 int elem2 = array2[i];

16 int result = elem1 + elem2;

17 // Create metadata with TagID=3 and Metadata=3 to ClientID=0

18 metadata_create(0,3,3);

19 array3[i] = result;

20 }

21

22/* Hardware Bounds Checker Functionality */

23HardwareBoundsChecker(CreateTagID, Address):

24 // Software communicates TagID using CREATE

25 TagIDRegister <= CreateTagID

26 // Bounds checker client performs a metadata lookup

27 // to find the TagID of address

28 MetadataTagID <= PerformMetadataLookup(Address)

29 // Interrupt rocket core if TagIDs do not match

30 // (i.e., access is out of bounds)

31 if MetadataTagID != TagIDRegister:

32 InterruptRocketCore()

Listing 2: MetaSys-based bounds checking example. Full source code is
available online [57].

####  6.1.2 Methodology and Evaluation

We evaluate MetaSys-based bounds checking on our prototype with the parameters
listed in Table 4 (tagging granularity is set to 64B). We use the Olden [130]
benchmarks (commonly used for bounds checking and stack protection research
[43, 174, 111, 146, 50] due to its focus on pointer-based data structures). We
only compare against a prior software solution [177] as custom hardware
solutions for bounds checking require intrusive changes to the
microarchitecture, ISA, and application, which are difficult to reasonably
implement on a full-system simulator or an FPGA. For example, implementing
Hardbound [43], the closest custom hardware solution for bounds checking,
requires compiler support, extending every register and word of memory with
“sidecar” shadow registers for base and bound addresses,
compression/decompression engines to compress these base/bound addresses, tag
caches, and bounds checking logic.

We evaluate three designs: _(i)_ the _Baseline_ system without MetaSys; _(ii)_
software bounds checking, based on prior work [177]; and _(iii)_ MetaSys-based
bounds checking. Fig. 5 depicts the execution time normalized to the Baseline.
We observe that the software bounds checking design incurs a high average
performance overhead of 36% (up to 82%). This overhead comes from executing
more instructions to check bounds (64% on average). In contrast, MetaSys-based
bounds checking incurs only an average performance overhead of 14% (up to
40%). MetaSys requires only a 32% increase in the number of executed
instructions. Workloads such as em3d, power, and mst, are highly compute
intensive and hence do not incur significant overheads with either bounds
checking technique.

Figure 5: Performance overheads for bounds checking

We conclude that MetaSys provides a lightweight substrate to implement and
evaluate hardware-software cooperative bounds checking. MetaSys can be
flexibly extended to implement more sophisticated memory protection
techniques.

###  6.2 Return Address Protection

The program’s call stack is a known source of many security vulnerabilities in
low-level, memory-unsafe languages such as C/C++. For example, the control
flow in the program can be hijacked by overwriting the _return addresses_
saved in the stack [36, 8]. Existing defenses such as ExecShield [160] and
stack canaries [36] do not protect against sophisticated attack techniques
[28, 24, 158]. Stack canary protection [36] is a software check that involves
writing an additional randomly-generated value in the stack and a duplicate is
saved separately in memory. The stack canary checks the randomly-generated
value in the stack against its duplicate to detect stack overwriting before a
function returns to the return address saved in the stack. Protecting return
addresses with more powerful software checks [16, 1, 97, 101, 83] incurs
significant runtime overheads and are hence difficult to use in practice [39,
156]. Prior work has proposed a range of hardware techniques [186, 86, 174,
129, 63, 45] to enable return address protection more efficiently. These
approaches either require dedicated hardware support for stack protection
(e.g., RAGuard [186], PAC-it-up [86], CET [63]) or more heavy-weight metadata
systems for memory protection (e.g., SDMP [129], Cheri [174], PUMP [45]). In
this section, we implement and evaluate return address protection with
MetaSys’s lightweight metadata support and cross-layer interfaces.

####  6.2.1 Return address protection with MetaSys

To enable return address protection with MetaSys, we first tag each return
address using MAP as id="1". This is done automatically with compiler support
and no programmer intervention is required. No CREATE instruction is used. In
hardware, we add support to simply disallow writes to any address tagged with
id="1". In order to do this, we implement a simple hardware optimization
client which is triggered by store instructions. For each store instruction,
the client performs a lookup to the MMC to determine the tag ID associated
with the address. If the tag="1", this indicates that the location is a return
address and the store is not allowed to complete. Any store to a tagged memory
address causes the hardware to invoke the OS to terminate the program. The
application can then unmap the return address when it is retrieved again from
the stack. This ensures that once a memory location within the stack has a
return address saved, it cannot be overwritten via attacks that hijack control
flow such as buffer overflow attacks.

####  6.2.2 Evaluation and Methodology.

We evaluate MetaSys-based return address protection using our FPGA prototype
with system parameters listed in Table 4 (the tagging granularity set to 64B).
We evaluate 3 designs using the Olden [130] benchmarks: _(i)_ the _Baseline_
system with no overheads; _(ii)_ canary stack protection [36] in the GCC
RISC-V compiler; and _(iii)_ MetaSys-based return address protection.

Fig. 6 depicts the execution time normalized to the _Baseline_. We observe
that the canary approach incurs a performance overhead of 5.5% (up to 20%),
while MetaSys incurs a diminished overhead of 1.2% (up to 6.2%). The major
overheads for the stack canaries come from executing extra instructions (5.5%
on average) to perform software checks. The overheads for MetaSys are low due
to the high MMC hit rate which leads to few additional memory accesses. In
addition to providing less overhead, MetaSys-based return address protection
can also protect against more sophisticated attacks that exploit write-what-
where gadgets [38] and, unlike canaries, are immune to information leaks
[151]. Protecting additional memory locations beyond return addresses (e.g.,
function pointers) with software approaches would incur even higher
instruction overhead. However, the observed MetaSys overhead would largely
remain the same as it already involves checking each store. In addition,
MetaSys-based return address protection utilizes the Metasys support and
interfaces, whose cost is amortized across many use cases, without requiring
specialized ISA and hardware support.

Figure 6: Performance overheads for return address protection

We conclude that MetaSys enables easy implementation and evaluation of
lightweight memory protection mechanisms with low performance overhead.

##  7 Other Use Cases of MetaSys

We briefly discuss various other cross-layer techniques that can be
implemented with MetaSys (but would be challenging to implement with prior
approaches like XMem [164]).

Performance optimization techniques. MetaSys provides a low-overhead framework
and a rich cross-layer interface to implement a diverse set of performance
optimizations including cache management, prefetching, page placement in
memory, approximation, data compression, DRAM cache management, and memory
management in NUMA and NUCA systems [66, 126, 58, 26, 167, 22, 137, 179, 136,
116, 117, 96, 159, 2, 41, 40]. MetaSys can flexibly implement the range of
cross-layer optimizations supported by XMem [164], and the Locality Descriptor
[163]. MetaSys’s _dynamic_ interface for metadata communication enables even
more powerful optimizations than XMem including memory optimizations for
dynamic data structures such as graphs. We already demonstrate one performance
optimization in §5.

Techniques to enforce cross-layer quality of service (QoS). MetaSys can be
used to implement cross-layer techniques to enforce QoS requirements of
applications in shared environments [93, 62, 107, 106, 76, 48, 153, 152].
MetaSys allows communicating an applications’ QoS requirements to hardware
components (e.g., the last-level cache, memory controllers) to enable
optimizations for partitioning and allocating shared resources such as cache
space and memory bandwidth.

Hardware support for debugging and monitoring. MetaSys can be used to
implement cross-layer techniques for performance debugging and bug detection
by providing efficient mechanisms to track memory access patterns using its
memory tagging and metadata lookup support. This includes efficient detection
of memory safety violations [161, 123] or concurrency bugs [89, 88, 108, 91,
90, 188] such as data races, deadlocks, or atomicity violations.

Security and protection. MetaSys provides a substrate to implement low-
overhead hardware techniques for security/protection: the tagged memory
support can be used to implement protection for spatial memory safety [42,
183, 172, 127], cache timing side-channels [78] and stack protection [129,
86]. For example, using MetaSys, software can tag memory accesses as security-
critical or safe. Based on the metadata received for every access, MetaSys can
activate/deactivate (for the specific access) the corresponding side-channel
defense technique at runtime (e.g., protect from or undo speculation [134,
178, 25, 17, 74]). We already demonstrate two security techniques in §6.

Garbage collection. MetaSys offers an efficient mechanism to track dead memory
regions, unreachable objects, or young objects in managed languages. MetaSys
is hence a natural substrate to implement hardware-software cooperative
approaches (such as prior work [69, 95, 94]) for garbage collection. For
example, HAMM [69], a hardware-software cooperative technique for reference
counting, tracks the number of references to any object in hardware. It has
many of the same metadata management components as MetaSys. HAMM uses a multi-
level metadata cache to manage the large amounts of metadata associated with
reference counting for each object. MetaSys was designed with modular
interfaces that enable adding more levels to the metadata cache for such use
cases.

OS optimizations. MetaSys can be used to implement OS optimizations that
require hardware performance monitoring of memory access patterns, contention,
reuse, etc [118, 105, 29, 148, 147, 47]. The metadata support in MetaSys can
be used to implement this monitoring and then inform OS optimizations like
thread scheduling, I/O scheduling, and page allocation/mapping [56, 77, 102,
40, 105].

Cache optimizations. MetaSys enables various cache optimizations such as cache
scrubbing [167, 136] and cache prioritization [66, 126, 58, 26, 167, 22, 137,
179, 136, 116, 117, 96, 159]. To implement such optimizations with MetaSys,
the CREATE operator is used to specify the expected _reuse_ of a data object
at runtime. For example, objects can be tagged as having no reuse (e.g., once
all threads have completed operations on it). Thus, upon encountering a cache
miss (the trigger event), the cache controller can look up the expected reuse
of different cache lines using MetaSys’s lookup mechanism and then evict the
dead cache line. A similar mechanism can be used to retain cache lines that
have high expected cache reuse.

Compressing sparse data structures. MetaSys can be used to support techniques
that efficiently compress sparse data structures and accelerate sparse
workloads [72, 138]. For example, SMASH [72] is a hardware-software
cooperative technique that efficiently compresses sparse matrices using a
hierarchy of bitmaps to encode non-zero cache lines and accelerates the
discovery of the non-zero elements of the sparse matrix. Instead of using
specialized hardware, SMASH could access the hierarchy of bitmaps and identify
non-zero elements with MetaSys’ metadata support.

Heterogeneous reliability memory optimizations. MetaSys’ metadata support can
be used by techniques that exploit heterogeneous reliability characteristics
of memory devices to improve performance, power consumption, and system cost
[81, 87, 135, 92]. These techniques typically require support for dynamically
looking up the error tolerance characteristics of data structures to place
them in memory to satisfy a target bit error rate. MetaSys’ metadata support
is a natural candidate for providing these techniques with a means to query
reliability characteristics of data structures.

##  8 Limitations of MetaSys

Our goal of providing a low-overhead and general system largely tailored for
cross-layer _performance_ optimization leads to several major limitations in
MetaSys. These limitations can be mitigated by future work.

Instruction and register tagging are not supported. MetaSys does not currently
support tagging of instructions or registers and thus cannot easily support
techniques such as taint-tracking [149, 37, 154] and security mechanisms that
require rule-checking at the instruction/register level [44, 45].

Overheads of fine-granularity memory tagging. While MetaSys supports memory
tagging at flexible granularity, the system is optimized for the _larger_
granularities typically required for performance optimization ($>=$64B) or
fine granularities for only _some_ data (e.g., return addresses). Byte/word
granularity tagging for the entire program data may lead to high MMC miss
rates and may thus incur higher overheads with MetaSys.

Limitations on using private metadata tables (PMTs) for runtime profiling.
With MetaSys’s existing interfaces, the PMTs cannot be used to collect program
information and supply it back to the application. The PMTs can only be
updated by the CREATE operator. This issue can be mitigated relatively easily
in future MetaSys versions.

##  9 Characterizing General Metadata Management Systems for Cross-Layer
Optimizations

Our goal in this section is to perform a detailed characterization of the
overheads of using a _single_ common metadata system and interface for
multiple cross-layer techniques. Three major challenges and sources of system
overhead include:

_(1) Handling dynamic metadata:_ Communicating metadata at runtime requires
execution of additional instructions in the program. This incurs performance
overheads in the form of CPU processing cycles and data movement to
communicate the metadata to hardware components or to save them in memory.

_(2) Efficient metadata management and lookups:_ The communicated metadata
must be saved in memory or specialized caches (the MMC in MetaSys) that
overflow to memory. Different components in the system must then be able to
efficiently look up the metadata for performance optimization. Storing and
retrieving metadata may incur expensive memory accesses and consume memory
bandwidth.

_(3) Scaling to multiple components:_ A _general_ cross-layer interface and
metadata system must be able to serve multiple client components implementing
different optimizations in the caches, the prefetchers, the memory controller,
etc. Multiple components accessing shared metadata support during program
execution poses significant scalability challenges.

The above challenges may impose significant area and performance overheads in
the CPU, making the feasibility of a common metadata system and interface (as
opposed to per-use-case specialized interfaces and optimizations) for cross-
layer techniques questionable. In this section, we set out to experimentally
quantify these overheads, identify key bottlenecks, and discover and provide
insights on how these challenges affect different workloads and how they can
be alleviated.

###  9.1 Analysis

We perform our characterization using the Polybench [124] and Ligra [145]
benchmark suites along with a set of microbenchmarks (available on Github
[57]). Polybench contains building block kernels frequently used in linear
algebra, scientific computation, and machine learning. Ligra contains widely
used graph analytics workloads. The microbenchmarks are designed to
_intensely_ stress the MetaSys system and identify _worst-case_ overheads.

  * •

_Stream._ This memory-bandwidth-intensive microbenchmark streams through a
large amount of data, accessing it only once. It hence has high spatial
locality and no data reuse.

  * •

_Linked List Traversal._ The microbenchmark mimics typical linked list
creation, insertion, and traversal and emulates the widely-seen memory-
intensive _pointer-chasing_ operation.

  * •

_Random Access._ This microbenchmark accesses memory locations within a large
array at random indices and is designed to test an extremely rare worst-case
scenario: no pattern in accesses, no reuse, and no spatial locality.

  * •

_3-dimensional array traversal (3D Array)._ This microbenchmark mimics the
access pattern and locality seen in applications with multi-dimensional
arrays. It traverses a 3D array first along the third dimension, and then
along the second and first (data is contiguously placed in the first
dimension). The access pattern is highly regular but exhibits no spatial
locality.

§4 describes the parameters of our baseline system. We summarize our key
findings in §9.3. In all evaluations in this section, since we aim to
characterize the overheads of the system itself, we do _not_ implement any
cross-layer optimization that improves performance. We simply implement
_lookups_ to the metadata system that an optimization could potentially make.
Since our goal is to stress the system and understand the worst-case
overheads, we perform metadata lookups for _every memory access_. In typical
use cases, the lookup triggers would be much less frequent than for every
memory access, e.g., lookups on only cache misses for prefetching or only
stores for return address protection.

####  9.1.1 Performance Overhead Analysis

The performance overheads in MetaSys come from two major sources: _(i)_
dynamic instructions (MAP and CREATE) and _(ii)_ metadata _lookups_ when a
component retrieves the tag ID associated with any memory address (from the
MMT, cached in the MMC) and then the corresponding metadata (in the PMT).

Fig. 7 depicts the execution time normalized to the baseline system (without
MetaSys) for two scenarios: _(i)_ when performing metadata lookups for _every_
access to the L1 cache (_All-Accesses_) and _(ii)_ when performing metadata
lookups only on every L1 cache miss (_Miss-Only_). These studies were
conducted with our baseline 128-entry MMC with a tagging granularity of 512B
(as in XMem [164]). Fig. 8 plots the number of memory accesses, normalized to
baseline (the additional memory accesses come from misses in the MMC). Fig. 9
plots the corresponding MMC hit rates.

Figure 7: Normalized performance overhead of MetaSys. Figure 8: Additional
memory accesses introduced by MetaSys metadata lookups. Figure 9: Metadata
Mapping Cache (MMC) hit rate.

We make two major observations from the three figures. First, the overall
performance overheads from the metadata management system for both designs are
low in most workloads with an average performance overhead of 2.7% (ranging
from $\sim$0% up to 14%), excluding the microbenchmarks. The highest overheads
observed in the microbenchmarks is 27% for Random and represents the worst-
case overhead. Workloads with the highest overheads (Random, GS, PR, TF) are
highly memory-intensive and have low spatial and temporal locality, which
leads to low hit rates in the MMC (e.g., $\sim$0% in Random and 24% in GS).
This causes a significant increase in accesses to memory and thus higher
performance overheads.

Second, the _number_ of metadata lookups (not shown in the figure) does not
have a direct impact on the overall performance overhead. _All-Access_
performs on average 75.2% more lookups than _Miss-Only_ , but incurs an
additional overhead of only 0.05%. _Miss-Only_ has lower MMC hit rates due to
lower locality in lookups than _All-Access_. Thus, the number of _additional_
memory accesses is largely the same for both designs, as shown in Fig. 8.

Since the major overheads are from additional memory accesses, we evaluate the
impact of available memory bandwidth. Fig. 10 depicts the performance overhead
of _All-Access_ on two different systems with 0.5$\times$ and 2$\times$ the
memory bandwidth of the baseline system. We observe that, except for GS, more
memory bandwidth significantly reduces any lookup overheads: the average
overhead is only 0.5% on a system with 2$\times$ the memory bandwidth.
Conversely, in workloads with higher MMC miss rates (e.g, DSYR2K), performance
overheads increase with a reduction in available memory bandwidth.

Figure 10: MetaSys performance overhead on systems with varying amounts of
memory bandwidth.

We conclude that _(i)_ performance overheads are correlated to the MMC hit
rates; _(ii)_ metadata lookup hardware can be frequently queried with no
direct observable impact on performance; and _(iii)_ overall performance
overheads are small when the MMC provides high hit rates.

Figure 11: Impact of Metadata Mapping Cache (MMC) size on MetaSys performance
overhead. Figure 12: Impact of tagging granularity on MetaSys performance
overhead.

####  9.1.2 Effect of the Metadata Mapping Cache (MMC)

Fig. 11 shows the impact of the size of the MMC on performance overhead. We
evaluate 6 sizes for _All-Access_ and Fig. 11 presents the resulting execution
time (normalized to the baseline system). We make two observations. First, in
most workloads, 128 entries is sufficient to obtain small overheads. This is
because with a 512B tagging granularity, we can hold tag IDs for 64KB of
memory in the MMC (compared to 16KB of L1 cache space). Second, workloads with
poor spatial and temporal locality (e.g., Random, GS), are largely insensitive
to the MMC sizes evaluated. Thus, overheads in such workloads cannot be easily
addressed by increasing the MMC size.

Figure 13: Impact of tagging granularity on additional TLB misses.

####  9.1.3 Effect of Metadata Granularity

The _granularity_ at which memory is tagged plays a critical role in
determining the reach of the MMC. Tagging memory at smaller granularities
requires more MMC entries to tag the same amount of memory, but it enables
more optimizations (e.g., bounds checking). Fig. 12 presents execution time
for different granularities of tagging, normalized to the baseline system
without MetaSys. For most workloads, even the smallest granularity we
evaluated (64B) has a small impact on performance. Large granularities reduce
overheads for all but Random and GS by significantly increasing the MMC hit
rate. A secondary effect in irregular workloads, such as PR and SSSP, is that
small granularities increase the number of _TLB_ misses (by 11% and 13%
respectively), as depicted in Fig. 13. The additional MMC misses cause
accesses to the MMT in memory which requires address translation.

To evaluate the effect of the TLB, we implement a MetaSys design that does
_not_ require address translation to access the MMT (i.e., MMT entries are
accessed directly using physical addresses). Fig. 14 presents the resulting
normalized execution time without address translation. We observe a decrease
in overhead with this design in the irregular workloads: BFS, CC, Random, and
LL (by 1.9%, 1.8%, 14%, and 1% respectively).

Figure 14: Performance overhead with no address translation overhead for
metadata.

####  9.1.4 Effect of Contention for Metadata

To evaluate the _scalability_ of MetaSys with multiple clients accessing the
same metadata support, we evaluate the overheads of two clients performing
frequent metadata lookups: one client on every _memory access_ (with the
corresponding memory address) and another on every _TLB miss_ (with the page
table entry address). Since each design performs lookups with _different_
memory addresses, they do not share entries in the MMC and this creates a
difficult scenario for the shared MMC.

Fig. 15 depicts the resulting execution time for two designs, normalized to
the baseline system: _(i)_ One Client performs metadata lookups on every
memory access and _(ii)_ Two Clients performs metadata lookups on every memory
access as well as on every TLB miss. We observe that for all workloads except
the microbenchmarks, increasing the number of clients leads to a small
additional performance overhead (on average 0.3%). This is because the MMC can
sufficiently capture the tag ID working set for both clients. The
microbenchmarks designed to stress the system experience a significant
additional performance overhead (up to 34% for Random) as a result of more
misses in the MMC due to more clients.

Figure 15: Performance overhead with multiple clients. Figure 16: Alleviating
MMC contention in microbenchmarks.

To investigate mechanisms to alleviate the MMC contention overheads seen in
the microbenchmarks, we evaluate three designs in Fig. 16: _(i)_
_Partitioning_ the MMC equally between the two clients; _(ii)_  _Prioritized
Insertion_ , where we insert mappings for the client with better locality at a
higher priority in the MMC (such that such mappings are evicted last); and
_(iii)_  _No stall_ , where we do not stall the core on an MMC miss (instead,
the optimization performed by the client is delayed). We observe that
_Partitioning_ reduces the overhead for 3D Array and LL by 9% and 4% by
avoiding cache thrashing. _Prioritized Insertion_ helps reduce the overheads
in LL (by 8.5%) and Random (by 6%), where one client has more locality than
the other in lookups. _No Stall_ significantly reduces the overhead in Random
(by 40%) by mitigating the latency overhead of additional memory accesses.

####  9.1.5 Evaluating instruction overheads

To evaluate the instruction overheads of the dynamic MAP/CREATE instructions,
we present an analysis in Fig. 17 where we intensively use these instructions:
_(i)_ for every eight memory instructions, we add one MAP and one CREATE
instructions and _(ii)_ for every two memory instructions, we add one MAP and
one CREATE instruction. We observe an average slowdown of 1.4% and 5.7%,
respectively over a baseline without MetaSys. This indicates that while
excessive use of MAP/CREATE instructions can lead to slowdowns, the primary
overheads are still from the metadata lookups which may lead to additional
memory accesses.

Figure 17: Performance overhead of MAP/CREATE instructions.

We conclude that MetaSys’ metadata support is scalable to multiple components
with small impact on performance overheads (except in microbenchmarks). The
overheads seen in microbenchmarks are a result of poor MMC hit rates that can
be mitigated via techniques such as partitioning, prioritized insertion, and
by not stalling the core on an MMC miss. Since optimizations are triggered by
loads/stores in MetaSys, MetaSys can be expected to gracefully scale to more
than two clients as most clients are expected to query the MMC using the same
addresses, which are aggregated and thus would not lead to additional lookups.

###  9.2 Hardware Area Overhead

We synthesized the baseline MetaSys system using the Synopsys DC [155] at 22nm
process technology to estimate the area overhead. MetaSys incurs small area
overhead: 0.03$mm^{2}$ (0.02% of a 22nm Intel Ivy Bridge CPU Core [144]).

###  9.3 Summary of Findings

(1) Despite stressing the metadata support, the overall performance overheads
of MetaSys are very low (2.7% on average, excluding the microbenchmarks). This
indicates that using metadata systems that are _general_ enough to support a
range of use cases is a promising approach to enabling cross-layer performance
optimizations in a general-purpose manner in real-world applications. The
higher overheads seen in microbenchmarks indicate that the worst-case
overheads are however substantially higher (up to 27%).

(2) Our studies indicate that MetaSys’ metadata management is _scalable_ to
support multiple client components that have high rates of metadata access
requirements. Performance overhead is dependent on locality of metadata
accesses as opposed to number of metadata accesses, indicating that the same
system can support multiple cross-layer optimizations at the same time. We
propose simple techniques to alleviate metadata contention generated by
multiple clients.

(3) The most critical factor that impacts MetaSys’ performance overhead is the
effectiveness of the Metadata Mapping Cache (MMC). Workloads with low locality
in metadata lookups incur performance overheads from additional memory
accesses. The _reach_ of the MMC is also affected by the _granularity_ at
which memory is tagged and hence the MMC hit rate can be improved with larger
granularities. Thus, efficient caching of metadata tags is critical.

(4) Accesses to metadata mappings in memory require address translation and
cause TLB misses, leading to high performance overhead especially in irregular
workloads (e.g., Random). We find that this overhead can be mitigated by using
physical addresses to access metadata mappings or by storing address
translations required to access metadata mappings in a separate TLB.

##  10 Conclusion

This work introduces MetaSys, the first open-source full-system FPGA-based
infrastructure to rapidly implement and evaluate diverse cross-layer
optimizations in real hardware. We demonstrate MetaSys’s versatility and ease-
of-use by implementing and evaluating three new cross-layer techniques. We
believe and hope MetaSys can enable new ideas and their rigorous evaluation on
real hardware.

Using MetaSys, we present the first detailed experimental characterization to
evaluate the efficiency and practicality of a single metadata system for
cross-layer performance optimization. We demonstrate that the associated
performance and area overheads are small, identify key performance
bottlenecks, and propose simple techniques to alleviate them. Our
characterization thus indicates that a _general_ hardware-software interface
with lightweight metadata management support offers a promising approach
towards enabling general-purpose cross-layer techniques in CPUs.

## Acknowledgements

We thank the anonymous reviewers of TACO, MICRO 2020, ISCA 2020/2021, and
SIGMETRICS 2020 for feedback. We thank the SAFARI group members for feedback
and the stimulating intellectual environment they provide. We acknowledge the
generous gifts provided by our industrial partners: Google, Huawei, Intel,
Microsoft, VMware, and the ETH Future Computing Laboratory.

## References

  * [1] M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti, “Control-flow Integrity,” in _CCS_ , 2005.
  * [2] N. Agarwal, D. Nellans, M. O’Connor, S. W. Keckler, and T. F. Wenisch, “Unlocking Bandwidth for GPUs in CC-NUMA Systems,” in _HPCA_ , 2015\.
  * [3] N. Agarwal, D. W. Nellans, M. Stephenson, M. O’Connor, and S. W. Keckler, “Page Placement Strategies for GPUs within Heterogeneous Memory Systems,” in _ASPLOS_ , 2015.
  * [4] J. Ahn, S. Hong, S. Yoo, O. Mutlu, and K. Choi, “A Scalable Processing-in-Memory Accelerator for Parallel Graph Processing,” in _ISCA_ , 2015.
  * [5] S. Ainsworth and T. M. Jones, “Graph Prefetching Using Data Structure Knowledge,” in _ICS_ , 2016.
  * [6] S. Ainsworth and T. M. Jones, “An event-triggered programmable prefetcher for irregular workloads,” in _ASPLOS_ , 2018.
  * [7] H. Al-Sukhni, I. Bratt, and D. A. Connors, “Compiler-Directed Content-Aware Prefetching for Dynamic Data Structures,” in _PACT_ , 2003.
  * [8] Aleph One, “Smashing The Stack For Fun And Profit,” 1996. [Online]. Available: https://inst.eecs.berkeley.edu/~cs161/fa08/papers/stack_smashing.pdf
  * [9] M. Annavaram, J. M. Patel, and E. S. Davidson, “Data Prefetching by Dependence Graph Precomputation,” in _ISCA_ , 2001.
  * [10] K. Asanović, R. Avizienis, J. Bachrach, S. Beamer, D. Biancolin, C. Celio, H. Cook, P. Dabbelt, J. R. Hauser, A. M. Izraelevitz, S. Karandikar, B. Keller, D. Kim, J. Koenig, Y. Lee, E. Love, M. Maas, A. Magyar, H. Mao, M. Moretó, A. Ou, D. A. Patterson, B. H. Richards, C. Schmidt, S. M. Twigg, H. Vo, and A. Waterman, “The Rocket Chip Generator,” Technical Report: UCB/EECS-2016-17, 2016.
  * [11] T. M. Austin, S. E. Breach, and G. S. Sohi, “Efficient Detection of All Pointer and Array Access Errors,” in _PLDI_ , 1994.
  * [12] AVNET, “Zynq®-7000 zedboard,” 2021. [Online]. Available: http://zedboard.org/product/zedboard
  * [13] J. Bachrach, H. Vo, B. Richards, Y. Lee, A. Waterman, R. Avižienis, J. Wawrzynek, and K. Asanović, “Chisel: Constructing Hardware in a Scala Embedded Language,” in _DAC_ , 2012.
  * [14] J.-L. Baer and T.-F. Chen, “An Effective On-chip Preloading Scheme to Reduce Data Access Penalty,” in _SC_ , 1991.
  * [15] M. Bakhshalipour, P. Lotfi-Kamran, and H. Sarbazi-Azad, “Domino Temporal Data Prefetcher,” in _HPCA_ , 2018.
  * [16] A. Baratloo, N. Singh, and T. Tsai, “Transparent Run-time Defense Against Stack Smashing Attacks,” in _ATEC_ , 2000.
  * [17] K. Barber, A. Bacha, L. Zhou, Y. Zhang, and R. Teodorescu, “Specshield: Shielding speculative data from microarchitectural covert channels,” in _PACT_ , 2019.
  * [18] A. Basak, S. Li, X. Hu, S. M. Oh, X. Xie, L. Zhao, X. Jiang, and Y. Xie, “Analysis and Optimization of the Memory Hierarchy for Graph Processing Workloads,” in _HPCA_ , 2019.
  * [19] M. Bekerman, S. Jourdan, R. Ronen, G. Kirshenboim, L. Rappoport, A. Yoaz, and U. Weiser, “Correlated Load-address Predictors,” in _ISCA_ , 1999.
  * [20] R. Bera, K. Kanellopoulos, A. Nori, T. Shahroodi, S. Subramoney, and O. Mutlu, “Pythia: A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning,” in _MICRO_ , 2021.
  * [21] R. Bera, A. V. Nori, O. Mutlu, and S. Subramoney, “DSPatch: Dual Spatial Pattern Prefetcher,” in _MICRO_ , 2019.
  * [22] K. Beyls and E. H. D’Hollander, “Generating Cache Hints for Improved Program Efficiency,” in _JSA_ , 2005.
  * [23] E. Bhatia, G. Chacon, S. Pugsley, E. Teran, P. V. Gratz, and D. A. Jiménez, “Perceptron-Based Prefetch Filtering,” in _ISCA_ , 2019.
  * [24] T. Bletsch, X. Jiang, V. Freeh, and Z. Liang, “Jump-oriented Programming: A New Class of Code-reuse Attack,” in _ASIA CCS_ , 2011.
  * [25] T. Bourgeat, I. Lebedev, A. Wright, S. Zhang, Arvind, and S. Devadas, “MI6: Secure Enclaves in a Speculative Out-of-Order Processor,” in _MICRO_ , 2009\.
  * [26] J. Brock, X. Gu, B. Bao, and C. Ding, “Pacman: Program-assisted Cache Management,” in _ISMM_ , 2013.
  * [27] N. P. Carter, S. W. Keckler, and W. J. Dally, “Hardware Support for Fast Capability-based Addressing,” in _ASPLOS_ , 1994.
  * [28] S. Checkoway, L. Davi, A. Dmitrienko, A.-R. Sadeghi, H. Shacham, and M. Winandy, “Return-oriented Programming Without Returns,” in _CCS_ , 2010\.
  * [29] T.-F. Chen and J.-L. Baer, “Effective Hardware-Based Data Prefetching for High-Performance Processors,” in _TC_ , 1995.
  * [30] Chi-Keung Luk and T. C. Mowry, “Cooperative Prefetching: Compiler and Hardware Support for Effective Instruction Prefetching in Modern Processors,” in _MICRO_ , 1998.
  * [31] T. M. Chilimbi and M. Hirzel, “Dynamic Hot Data Stream Prefetching for General-Purpose Programs,” in _PLDI_ , 2002.
  * [32] T. Chiueh, “Sunder: A Programmable Hardware Prefetch Architecture for Numerical Loops,” in _SC_ , 1994.
  * [33] Y. Chou, “Low-cost Epoch-based Correlation Prefetching for Commercial Applications,” in _MICRO_ , 2007.
  * [34] J. Condit, M. Harren, Z. Anderson, D. Gay, and G. C. Necula, “Dependent Types for Low-level Programming,” in _ESOP_ , 2007.
  * [35] R. Cooksey, S. Jourdan, and D. Grunwald, “A Stateless, Content-directed Data Prefetching Mechanism,” in _ASPLOS_ , 2002.
  * [36] C. Cowan, C. Pu, D. Maier, H. Hintony, J. Walpole, P. Bakke, S. Beattie, A. Grier, P. Wagle, and Q. Zhang, “StackGuard: Automatic Adaptive Detection and Prevention of Buffer-overflow Attacks,” in _USENIX Security_ , 1998\.
  * [37] J. R. Crandall, S. F. Wu, and F. T. Chong, “Minos: Architectural Support for Protecting Control Data,” in _TACO_ , 2006.
  * [38] CWE MITRE, “CWE-123: Write-what-where Condition,” 2019. [Online]. Available: https://cwe.mitre.org/data/definitions/123.html
  * [39] T. H. Dang, P. Maniatis, and D. Wagner, “The Performance Cost of Shadow Stacks and Stack Canaries,” in _ASIA CCS_ , 2015.
  * [40] R. Das, R. Ausavarungnirun, O. Mutlu, A. Kumar, and M. Azimi, “Application-to-core Mapping Policies to Reduce Memory System Interference in Multi-core Systems,” in _HPCA_ , 2013.
  * [41] M. Dashti, A. Fedorova, J. Funston, F. Gaud, R. Lachaize, B. Lepers, V. Quéma, and M. Roth, “Traffic Management: A Holistic Approach to Memory Placement on NUMA Systems,” in _ASPLOS_ , 2013.
  * [42] D. Y. Deng and G. E. Suh, “High-performance Parallel Accelerator for Flexible and Efficient Run-time Monitoring,” in _DSN_ , 2012.
  * [43] J. Devietti, C. Blundell, M. M. K. Martin, and S. Zdancewic, “Hardbound: Architectural Support for Spatial Safety of the C Programming Language,” in _ASPLOS_ , 2008.
  * [44] U. Dhawan, C. Hritcu, R. Rubin, N. Vasilakis, S. Chiricescu, J. M. Smith, T. F. Knight, Jr., B. C. Pierce, and A. DeHon, “Architectural support for software-defined metadata processing,” in _ASPLOS_ , 2015.
  * [45] U. Dhawan, N. Vasilakis, R. Rubin, S. Chiricescu, J. M. Smith, T. F. Knight, Jr., B. C. Pierce, and A. DeHon, “PUMP: A Programmable Unit for Metadata Processing,” in _HASP_ , 2014.
  * [46] D. Dhurjati and V. Adve, “Backwards-compatible Array Bounds Checking for C with Very Low Overhead,” in _ICSE_ , 2006.
  * [47] S. R. Dulloor, A. Roy, Z. Zhao, N. Sundaram, N. Satish, R. Sankaran, J. Jackson, and K. Schwan, “Data Tiering in Heterogeneous Memory Systems,” in _EuroSys_ , 2016.
  * [48] E. Ebrahimi, C. J. Lee, O. Mutlu, and Y. N. Patt, “Fairness via Source Throttling: A Configurable and High-performance Fairness Substrate for Multi-core Memory Systems,” in _ASPLOS_ , 2010.
  * [49] E. Ebrahimi, O. Mutlu, and Y. N. Patt, “Techniques for Bandwidth-efficient Prefetching of Linked Data Structures in Hybrid Prefetching Systems,” in _HPCA_ , 2009.
  * [50] A. S. Elliott, A. Ruef, M. Hicks, and D. Tarditi, “Checked C: Making C Safe by Extension,” in _SecDev_ , 2018.
  * [51] M. Ferdman and B. Falsafi, “Last-touch Correlated Data Streaming,” in _ISPASS_ , 2007.
  * [52] M. Ferdman, S. Somogyi, and B. Falsafi, “Spatial Memory Streaming with Rotated Patterns,” in _In 1st JILP Data Prefetching Championship_ , 2009\.
  * [53] E. A. Feustel, “On The Advantages of Tagged Architecture,” in _TC_ , 1973\.
  * [54] J. W. C. Fu and J. H. Patel, “Data Prefetching in Multiprocessor Vector Cache Memories,” in _ISCA_ , 1991.
  * [55] J. W. C. Fu, J. H. Patel, and B. L. Janssens, “Stride Directed Prefetching in Scalar Processors,” in _MICRO_ , 1992.
  * [56] M. Gorman, “Physical Page Allocation,” 2007. [Online]. Available: https://www.kernel.org/doc/gorman/html/understand/understand009.html
  * [57] S. R. Group, “MetaSys,” 2021. [Online]. Available: https://github.com/CMU-SAFARI/MetaSys
  * [58] X. Gu, T. Bai, Y. Gao, C. Zhang, R. Archambault, and C. Ding, “P-OPT: Program-Directed Optimal Cache Management,” in _LCPC_ , 2008.
  * [59] M. Hashemi, K. Swersky, J. Smith, G. Ayers, H. Litz, J. Chang, C. Kozyrakis, and P. Ranganathan, “Learning Memory Access Patterns,” in _ICML_ , 2018\.
  * [60] R. Hastings and B. Joyce, “Purify: Fast Detection of Memory Leaks and Access Errors,” in _USENIX_ , 1991.
  * [61] Z. Hu, M. Martonosi, and S. Kaxiras, “TCP: Tag Correlating Prefetchers,” in _HPCA_ , 2003.
  * [62] B. Huang, X. Jin, H. Wang, Y. Zhou, Z. Chang, Y. Cao, and Y. Bao, “Labeled RISC-V: A New Perspective on Software-Defined Architecture,” in _CARVV_ , 2017.
  * [63] Intel, “Control-flow Enforcement Technology Specification,” 2019. [Online]. Available: www.intel.com/content/dam/www/public/us/en/documents/white-papers/virtualization-enabling-intel-virtualization-technology-features-and-benefits-paper.pdf
  * [64] Y. Ishii, M. Inaba, and K. Hiraki, “Access Map Pattern Matching for Data Cache Prefetch,” in _ISC_ , 2009.
  * [65] A. Jain and C. Lin, “Linearizing Irregular Memory Accesses for Improved Correlated Prefetching,” in _MICRO_ , 2013.
  * [66] P. Jain, S. Devadas, D. Engels, and L. Rudolph, “Software-Assisted Cache Replacement Mechanisms for Embedded Systems,” in _ICCAD_ , 2001.
  * [67] T. Jim, J. G. Morrisett, D. Grossman, M. W. Hicks, J. Cheney, and Y. Wang, “Cyclone: A Safe Dialect of C,” in _ATEC_ , 2002.
  * [68] A. Joannou, J. Woodruff, R. Kovacsics, S. W. Moore, A. Bradbury, H. Xia, R. N. M. Watson, D. Chisnall, M. Roe, B. Davis, E. Napierala, J. Baldwin, K. Gudka, P. G. Neumann, A. Mazzinghi, A. Richardson, S. Son, and A. T. Markettos, “Efficient Tagged Memory,” in _ICCD_ , 2017.
  * [69] J. A. Joao, O. Mutlu, and Y. N. Patt, “Flexible Reference-Counting-Based Hardware Acceleration for Garbage Collection,” in _ISCA_ , 2009.
  * [70] D. Joseph and D. Grunwald, “Prefetching using Markov predictors,” in _ISCA_ , 1997.
  * [71] N. P. Jouppi, “Improving Direct-mapped Cache Performance by the Addition of a Small Fully-associative Cache and Prefetch Buffers,” in _ISCA_ , 1990.
  * [72] K. Kanellopoulos, N. Vijaykumar, C. Giannoula, R. Azizi, S. Koppula, N. M. Ghiasi, T. Shahroodi, J. G. Luna, and O. Mutlu, “SMASH: Co-Designing Software Compression and Hardware-Accelerated Indexing for Efficient Sparse Matrix Operations,” in _MICRO_ , 2019.
  * [73] M. Karlsson, F. Dahlgren, and P. Stenstrom, “A Prefetching Technique for Irregular Accesses to Linked Data Structures,” in _HPCA_ , 2000.
  * [74] K. N. Khasawneh, E. M. Koruyeh, C. Song, D. Evtyushkin, D. Ponomarev, and N. Abu-Ghazaleh, “SafeSpec: Banishing the Spectre of a Meltdown with Leakage-Free Speculation,” in _DAC_ , 2019.
  * [75] J. Kim, S. H. Pugsley, P. V. Gratz, A. Reddy, C. Wilkerson, and Z. Chishti, “Path Confidence based Lookahead Prefetching,” in _MICRO_ , 2016.
  * [76] Y. Kim, M. Papamichael, O. Mutlu, and M. Harchol-Balter, “Thread Cluster Memory Scheduling: Exploiting Differences in Memory Access Behavior,” in _MICRO_ , 2010.
  * [77] K. C. Knowlton, “A Fast Storage Allocator,” in _CACM_ , 1965.
  * [78] P. C. Kocher, “Timing Attacks on Implementations of Diffie-Hellman, RSA, DSS, and Other Systems,” in _CRYPTO_ , 1996.
  * [79] C. Kolbitsch, C. Kruegel, and E. Kirda, “Extending mondrian memory protection,” 2011. [Online]. Available: https://www.sto.nato.int/publications/STO%20Meeting%20Proceedings/RTO-MP-IST-091/MP-IST-091-10.pdf
  * [80] S. Kondguli and M. Huang, “Division of labor: A more effective approach to prefetching,” in _ISCA_ , 2018.
  * [81] S. Koppula, L. Orosa, A. G. Yağlıkçı, R. Azizi, T. Shahroodi, K. Kanellopoulos, and O. Mutlu, “EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM,” in _MICRO_ , 2019.
  * [82] S. Kumar and C. Wilkerson, “Exploiting Spatial Locality in Data Caches using Spatial Footprints,” in _ISCA_ , 1998.
  * [83] V. Kuznetsov, L. Szekeres, M. Payer, G. Candea, R. Sekar, and D. Song, “Code-Pointer Integrity,” in _OSDI_ , 2014.
  * [84] A. Kwon, U. Dhawan, J. M. Smith, T. F. Knight, Jr., and A. DeHon, “Low-fat Pointers: Compact Encoding and Efficient Gate-level Implementation of Fat Pointers for Spatial Safety and Capability-based Security,” in _CCS_ , 2013\.
  * [85] H. M. Levy, _Capability-based Computer Systems_ , 1984.
  * [86] H. Liljestrand, T. Nyman, K. Wang, C. C. Perez, J.-E. Ekberg, and N. Asokan, “PAC it up: Towards Pointer Integrity using ARM Pointer Authentication,” in _USENIX Security_ , 2018.
  * [87] S. Liu, K. Pattabiraman, T. Moscibroda, and B. G. Zorn, “Flikker: Saving DRAM Refresh-power Through Critical Data Partitioning,” in _ASPLOS_ , 2011\.
  * [88] S. Lu, J. Tucek, F. Qin, and Y. Zhou, “AVIO: Detecting Atomicity Violations via Access Interleaving Invariants,” in _ASPLOS_ , 2006.
  * [89] B. Lucia, L. Ceze, and K. Strauss, “ColorSafe: Architectural Support for Debugging and Dynamically Avoiding Multi-variable Atomicity Violations,” in _ISCA_ , 2010.
  * [90] B. Lucia, L. Ceze, K. Strauss, S. Qadeer, and H.-J. Boehm, “Conflict Exceptions: Simplifying Concurrent Language Semantics with Precise Hardware Exceptions for Data-races,” in _ISCA_ , 2010.
  * [91] B. Lucia, J. Devietti, K. Strauss, and L. Ceze, “Atom-aid: Detecting and Surviving Atomicity Violations,” in _ISCA_ , 2008.
  * [92] Y. Luo, S. Govindan, B. Sharma, M. Santaniello, J. Meza, A. Kansal, J. Liu, B. Khessib, K. Vaid, and O. Mutlu, “Characterizing Application Memory Error Vulnerability to Optimize Datacenter Cost via Heterogeneous-reliability Memory,” in _DSN_ , 2014.
  * [93] J. Ma, X. Sui, N. Sun, Y. Li, Z. Yu, B. Huang, T. Xu, Z. Yao, Y. Chen, H. Wang, L. Zhang, and Y. Bao, “Supporting Differentiated Services in Computers via Programmable Architecture for Resourcing-on-Demand (PARD),” in _ASPLOS_ , 2015.
  * [94] M. Maas, K. Asanovic, and J. Kubiatowicz, “Grail Quest: A New Proposal for Hardware-assisted Garbage Collection,” in _Workshop on Architectures and Systems for Big Data_ , 2016.
  * [95] M. Maas, K. Asanović, and J. Kubiatowicz, “A Hardware Accelerator for Tracing Garbage Collection,” in _ISCA_ , 2018.
  * [96] M. Manivannan, V. Papaefstathiou, M. Pericas, and P. Stenstrom, “RADAR: Runtime-Assisted Dead Region Management for Last-Level Caches,” in _HPCA_ , 2016.
  * [97] A. J. Mashtizadeh, A. Bittau, D. Boneh, and D. Mazières, “CCFI: Cryptographically Enforced Control Flow Integrity,” in _CCS_ , 2015.
  * [98] A. Menon, S. Murugan, C. Rebeiro, N. Gala, and K. Veezhinathan, “Shakti-T: A RISC-V Processor with Light Weight Security Extensions,” in _HASP_ , 2017.
  * [99] P. Michaud, “Best-offset Hardware Prefetching,” in _HPCA_ , 2016.
  * [100] D. Midi, M. Payer, and E. Bertino, “Memory Safety for Embedded Devices with nesCheck,” in _ASIA CCS_ , 2017.
  * [101] V. Mohan, P. Larsen, S. Brunthaler, K. W. Hamlen, and M. Franz, “Opaque Control-Flow Integrity,” in _NDSS_ , 2015.
  * [102] I. Molnar, “Modular Scheduler Core and Completely Fair Scheduler,” 2007. [Online]. Available: https://web.archive.org/web/20070419102054/http://kerneltrap.org/node/8059
  * [103] A. Mukkara, N. Beckmann, M. Abeydeera, X. Ma, and D. Sanchez, “Exploiting Locality in Graph Analytics Through Hardware-accelerated Traversal Scheduling,” in _MICRO_ , 2018.
  * [104] A. Mukkara, N. Beckmann, and D. Sanchez, “Whirlpool: Improving Dynamic Cache Management with Static Data Classification,” in _ASPLOS_ , 2016.
  * [105] Muralidhara, L. Subramanian, O. Mutlu, M. Kandemir, and T. Moscibroda, “Reducing Memory Interference in Multicore Systems via Application-aware Memory Channel Partitioning,” in _MICRO_ , 2011.
  * [106] O. Mutlu and T. Moscibroda, “Stall-Time Fair Memory Access Scheduling for Chip Multiprocessors,” in _MICRO_ , 2007.
  * [107] O. Mutlu and T. Moscibroda, “Parallelism-Aware Batch Scheduling: Enhancing both Performance and Fairness of Shared DRAM Systems,” in _ISCA_ , 2008\.
  * [108] A. Muzahid, N. Otsuki, and J. Torrellas, “AtomTracker: A Comprehensive Approach to Atomic Region Inference and Violation Detection,” in _MICRO_ , 2010.
  * [109] S. Nagarakatte, M. M. K. Martin, and S. Zdancewic, “Watchdog: Hardware for Safe and Secure Manual Memory Management and Full Memory Safety,” in _ISCA_ , 2012.
  * [110] S. Nagarakatte, M. M. K. Martin, and S. Zdancewic, “WatchdogLite: Hardware-Accelerated Compiler-Based Pointer Checking,” in _CGO_ , 2014.
  * [111] S. Nagarakatte, J. Zhao, M. M. Martin, and S. Zdancewic, “SoftBound: Highly Compatible and Complete Spatial Memory Safety for C,” in _PLDI_ , 2009.
  * [112] G. C. Necula, J. Condit, M. Harren, S. McPeak, and W. Weimer, “CCured: Type-safe Retrofitting of Legacy Software,” in _TOPLAS_ , 2005.
  * [113] K. Nilakant, V. Dalibard, A. Roy, and E. Yoneki, “PrefEdge: SSD Prefetcher for Large-Scale Graph Traversal,” in _SYSTOR_ , 2014.
  * [114] O. Oleksenko, D. Kuvaiskii, P. Bhatotia, P. Felber, and C. Fetzer, “Intel MPX Explained: A Cross-layer Analysis of the Intel MPX System Stack,” in _SIGMETRICS_ , 2018.
  * [115] S. Pakalapati and B. Panda, “Bouquet of Instruction Pointers: Instruction Pointer Classifier-based Spatial Hardware Prefetching,” in _ISCA_ , 2020\.
  * [116] A. Pan and V. S. Pai, “Runtime-driven Shared Last-level Cache Management for Task-parallel Programs,” in _SC_ , 2015.
  * [117] V. Papaefstathiou, M. G. Katevenis, D. S. Nikolopoulos, and D. Pnevmatikatos, “Prefetching and Cache Management Using Task Lifetimes,” in _ICS_ , 2013\.
  * [118] H. Park, S. Baek, J. Choi, D. Lee, and S. H. Noh, “Regularities Considered Harmful: Forcing Randomness to Memory Accesses to Reduce Row Buffer Conflicts for Multi-core, Multi-bank Systems,” in _ASPLOS_ , 2013.
  * [119] H. Patil and C. N. Fischer, “Efficient Run-time Monitoring Using Shadow Processing,” in _AADEBUG_ , 1995.
  * [120] L. Peled, S. Mannor, U. Weiser, and Y. Etsion, “Semantic Locality and Context-based Prefetching using Reinforcement Learning,” in _ISCA_ , 2015\.
  * [121] L. Peled, S. Mannor, U. Weiser, and Y. Etsion, “Semantic Locality and Context-based Prefetching Using Reinforcement Learning,” in _ISCA_ , 2015\.
  * [122] L. Peled, U. Weiser, and Y. Etsion, “A Neural Network Memory Prefetcher Using Semantic Locality,” arXiv:1804.00478, 2018.
  * [123] Pin Zhou, Feng Qin, Wei Liu, Yuanyuan Zhou, and J. Torrellas, “iWatcher: Efficient Architectural Support for Software Debugging,” in _ISCA_ , 2004.
  * [124] L. Pouchet, “Polybench: The polyhedral benchmark suite,” 2015. [Online]. Available: http://web.cse.ohio-state.edu/~pouchet.2/software/polybench/
  * [125] S. H. Pugsley, Z. Chishti, C. Wilkerson, P.-f. Chuang, R. L. Scott, A. Jaleel, S.-L. Lu, K. Chow, and R. Balasubramonian, “Sandbox Prefetching: Safe Run-time Evaluation of Aggressive Prefetchers,” in _HPCA_ , 2014.
  * [126] R. Ravindran, M. Chu, and S. Mahlke, “Compiler-managed Partitioned Data Caches for Low Power,” in _LCTES_ , 2007.
  * [127] S. K. Reinhardt, J. R. Larus, and D. A. Wood, “Tempest and Typhoon: User-level Shared Memory,” in _ISCA_ , 1994.
  * [128] RISC-V, “RISC-V proxy kernel,” 2019. [Online]. Available: https://github.com/riscv/riscv-pk
  * [129] N. Roessler and A. Dehon, “Protecting the Stack with Metadata Policies and Tagged Hardware,” in _SP_ , 2018.
  * [130] A. Rogers, M. C. Carlisle, J. H. Reppy, and L. J. Hendren, “Supporting Dynamic Data Structures on Distributed-Memory Machines,” in _TOPLAS_ , 1995\.
  * [131] A. Roth, A. Moshovos, and G. S. Sohi, “Dependence Based Prefetching for Linked Data Structures,” in _ASPLOS_ , 1998.
  * [132] A. Roth and G. S. Sohi, “Effective Jump-Pointer Prefetching for Linked Data Structures,” in _ISCA_ , 1999.
  * [133] O. Ruwase and M. S. Lam, “A Practical Dynamic Buffer Overflow Detector,” in _NDSS_ , 2004.
  * [134] G. Saileshwar and M. K. Qureshi, “CleanupSpec: An “Undo” Approach to Safe Speculation,” in _MICRO_ , 2019.
  * [135] A. Sampson, J. Nelson, K. Strauss, and L. Ceze, “Approximate Storage in Solid-state Memories,” in _TOCS_ , 2014.
  * [136] J. B. Sartor, W. Heirman, S. M. Blackburn, L. Eeckhout, and K. S. McKinley, “Cooperative Cache Scrubbing,” in _PACT_ , 2014.
  * [137] J. B. Sartor, S. Venkiteswaran, K. S. McKinley, and Z. Wang, “Cooperative Caching with Keep-me and Evict-me,” in _INTERACT_ , 2005.
  * [138] V. Seshadri, G. Pekhimenko, O. Ruwase, O. Mutlu, P. B. Gibbons, M. A. Kozuch, T. C. Mowry, and T. Chilimbi, “Page Overlays: An Enhanced Virtual Memory Framework to Enable Fine-Grained Memory Management,” in _ISCA_ , 2015.
  * [139] M. Shakerinava, M. Bakhshalipour, P. Lotfi-Kamran, and H. Sarbazi-Azad, “Multi-Lookahead Offset Prefetching,” 3rd Data Prefetching Championship, 2019\.
  * [140] M. Shevgoor, S. Koladiya, R. Balasubramonian, C. Wilkerson, S. H. Pugsley, and Z. Chishti, “Efficiently prefetching complex address patterns,” in _MICRO_ , 2015.
  * [141] Z. Shi, A. Jain, K. Swersky, M. Hashemi, P. Ranganathan, and C. Lin, “A Neural Hierarchical Sequence Model for Irregular Data Prefetching,” in _ML For Systems Workshop, NeurIPS_ , 2019.
  * [142] Z. Shi, A. Jain, K. Swersky, M. Hashemi, P. Ranganathan, and C. Lin, “A Hierarchical Neural Model of Data Prefetching,” in _ASPLOS_ , 2021.
  * [143] Z. Shi, K. Swersky, D. Tarlow, P. Ranganathan, and M. Hashemi, “Learning Execution Through Neural Code Fusion,” arXiv:1906.07181, 2019.
  * [144] A. L. Shimpi, “Dual Core/GT2 Ivy Bridge Die Measured: 121mm2,” 2012. [Online]. Available: https://www.anandtech.com/show/5875/dual-coregt2-ivy-bridge-die-measured-121mm2
  * [145] J. Shun and G. E. Blelloch, “Ligra: A Lightweight Graph Processing Framework for Shared Memory,” in _PPoPP_ , 2013.
  * [146] M. S. Simpson and R. K. Barua, “MemSafe: Ensuring the Spatial and Temporal Memory Safety of C at Runtime,” in _SCAM_ , 2010.
  * [147] S. Somogyi, T. F. Wenisch, A. Ailamaki, and B. Falsafi, “Spatio-Temporal Memory Streaming,” in _ISCA_ , 2009.
  * [148] S. Somogyi, T. F. Wenisch, A. Ailamaki, B. Falsafi, and A. Moshovos, “Spatial Memory Streaming,” in _ISCA_ , 2006.
  * [149] C. Song, H. Moon, M. Alam, I. Yun, B. Lee, T. Kim, W. Lee, and Y. Paek, “HDFI: Hardware-Assisted Data-Flow Isolation,” in _SP_ , 2016.
  * [150] S. Srinath, O. Mutlu, H. Kim, and Y. N. Patt, “Feedback Directed Prefetching: Improving the Performance and Bandwidth-efficiency of Hardware Prefetchers,” in _HPCA_ , 2007.
  * [151] R. Strackx, Y. Younan, P. Philippaerts, F. Piessens, S. Lachmund, and T. Walter, “Breaking the Memory Secrecy Assumption,” in _EUROSEC_ , 2009\.
  * [152] L. Subramanian, V. Seshadri, A. Ghosh, S. Khan, and O. Mutlu, “The Application Slowdown Model: Quantifying and Controlling the Impact of Inter-Application Interference at Shared Caches and Main Memory,” in _MICRO_ , 2015.
  * [153] L. Subramanian, V. Seshadri, Y. Kim, B. Jaiyen, and O. Mutlu, “MISE: Providing Performance Predictability and Improving Fairness in Shared Main Memory Systems,” in _HPCA_ , 2013.
  * [154] G. E. Suh, J. W. Lee, D. Zhang, and S. Devadas, “Secure Program Execution via Dynamic Information Flow Tracking,” in _ASPLOS_ , 2004.
  * [155] Synopsys, “Synopsys Design Compiler,” 2021. [Online]. Available: https://www.synopsys.com/support/training/rtl-synthesis/design-compiler-rtl-synthesis.html
  * [156] L. Szekeres, M. Payer, T. Wei, and D. Song, “SoK: Eternal War in Memory,” in _SP_ , 2013.
  * [157] N. Talati, K. May, A. Behroozi, Y. Yang, K. Kaszyk, C. Vasiladiotis, T. Verma, L. Li, B. Nguyen, J. Sun, J. M. Morton, A. Ahmadi, T. Austin, M. O’Boyle, S. Mahlke, T. Mudge, and R. Dreslinski, “Prodigy: Improving the Memory Latency of Data-Indirect Irregular Workloads Using Hardware-Software Co-Design,” in _HPCA_ , 2021.
  * [158] M. Tran, M. Etheridge, T. Bletsch, X. Jiang, V. Freeh, and P. Ning, “On the Expressiveness of Return-into-libc Attacks,” in _RAID_ , 2011.
  * [159] G. Tyson, M. Farrens, J. Matthews, and A. R. Pleszkun, “A Modified Approach to Data Cache Management,” in _MICRO_ , 1995.
  * [160] A. van de Ven, “New Security Enhancements in Red Hat Enterprise Linux: v.3 update 3,” 2004. [Online]. Available: https://static.redhat.com/legacy/f/pdf/rhel/WHP0006US_Execshield.pdf
  * [161] G. Venkataramani, B. Roemer, Y. Solihin, and M. Prvulovic, “MemTracker: Efficient and Programmable Support for Memory Access Monitoring and Debugging,” in _HPCA_ , 2007.
  * [162] N. Vijaykumar, “Enhancing Programmability, Portability, and Performance with Rich Cross-Layer Abstractions,” Ph.D. dissertation, 2019.
  * [163] N. Vijaykumar, E. Ebrahimi, K. Hsieh, P. B. Gibbons, and O. Mutlu, “The Locality Descriptor: A Holistic Cross-Layer Abstraction to Express Data Locality in GPUs,” in _ISCA_ , 2018.
  * [164] N. Vijaykumar, A. Jain, D. Majumdar, K. Hsieh, G. Pekhimenko, E. Ebrahimi, N. Hajinazar, P. B. Gibbons, and O. Mutlu, “A Case for Richer Cross-layer Abstractions: Bridging the Semantic Gap with Expressive Memory,” in _ISCA_ , 2018.
  * [165] Z. Wang and T. Nowatzki, “Stream-based Memory Access Specialization for General Purpose Processors,” in _ISCA_ , 2019.
  * [166] Z. Wang, K. S. Mckinley, and D. Burger, “Combining Cooperative Software/Hardware Prefetching and Cache Replacement,” in _IBM Austin CAS Center for Advanced Studies Conference_ , 2004.
  * [167] Z. Wang, K. S. McKinley, A. L. Rosenberg, and C. C. Weems, “Using the Compiler to Improve Cache Replacement Decisions,” in _PACT_ , 2002.
  * [168] R. N. M. Watson, J. Anderson, B. Laurie, and K. Kennaway, “Capsicum: Practical Capabilities for UNIX,” in _USENIX Security_ , 2010.
  * [169] T. F. Wenisch, M. Ferdman, A. Ailamaki, B. Falsafi, and A. Moshovos, “Practical Off-chip Meta-data for Temporal Memory Streaming,” in _HPCA_ , 2009.
  * [170] T. F. Wenisch, M. Ferdman, A. Ailamaki, B. Falsafi, and A. Moshovos, “Making Address-correlated Prefetching Practical,” in _IEEE Micro_ , 2010.
  * [171] T. F. Wenisch, S. Somogyi, N. Hardavellas, J. Kim, A. Ailamaki, and B. Falsafi, “Temporal streaming of shared memory,” in _ISCA_ , 2005.
  * [172] A. Wiggins, S. Winwood, H. Tuch, and G. Heiser, “Legba: Fast Hardware Support for Fine-Grained Protection,” in _ASIA CCS_ , 2003.
  * [173] E. Witchel, J. Cates, and K. Asanović, “Mondrian Memory Protection,” in _ASPLOS_ , 2002.
  * [174] J. Woodruff, R. N. Watson, D. Chisnall, S. W. Moore, J. Anderson, B. Davis, B. Laurie, P. G. Neumann, R. Norton, and M. Roe, “The CHERI Capability Model: Revisiting RISC in an Age of Risk,” in _ISCA_ , 2014.
  * [175] H. Wu, K. Nathella, J. Pusdesris, D. Sunwoo, A. Jain, and C. Lin, “Temporal Prefetching Without the Off-Chip Metadata,” in _MICRO_ , 2019.
  * [176] H. Wu, K. Nathella, D. Sunwoo, A. Jain, and C. Lin, “Efficient Metadata Management for Irregular Data Prefetching,” in _ISCA_ , 2019.
  * [177] W. Xu, D. C. DuVarney, and R. Sekar, “An Efficient and Backwards-compatible Transformation to Ensure Memory Safety of C Programs,” in _SIGSOFT_ , 2004\.
  * [178] M. Yan, J. Choi, D. Skarlatos, A. Morrison, C. W. Fletcher, and J. Torrellas, “InvisiSpec: Making Speculative Execution Invisible in the Cache Hierarchy,” in _MICRO_ , 2018.
  * [179] H. Yang, R. Govindarajan, G. R. Gao, and Z. Hu, “Compiler-Assisted Cache Replacement: Problem Formulation and Performance Evaluation,” in _LCPC_ , 2003.
  * [180] S. H. Yong and S. Horwitz, “Protecting C Programs from Attacks via Invalid Pointer Dereferences,” in _ESEC_ , 2003.
  * [181] X. Yu, C. J. Hughes, N. Satish, and S. Devadas, “IMP: Indirect Memory Prefetcher,” in _MICRO_ , 2015.
  * [182] N. Zeldovich, H. Kannan, M. Dalton, and C. Kozyrakis, “Hardware enforcement of application security policies using tagged memory,” in _OSDI_ , 2008.
  * [183] N. Zeldovich, H. Kannan, M. Dalton, and C. Kozyrakis, “Hardware Enforcement of Application Security Policies Using Tagged Memory,” in _OSDI_ , 2008\.
  * [184] Y. Zeng and X. Guo, “Long Short Term Memory based Hardware Prefetcher: A Case Study,” in _MEMSYS_ , 2017.
  * [185] D. Zhang, X. Ma, M. Thomson, and D. Chiou, “Minnow: Lightweight Offload Engines for Worklist Management and Worklist-Directed Prefetching,” in _ASPLOS_ , 2018.
  * [186] J. Zhang, R. Hou, J. Fan, K. Liu, L. Zhang, and S. A. McKee, “RAGuard: A Hardware Based Mechanism for Backward-Edge Control-Flow Integrity,” in _CF_ , 2017.
  * [187] Zhenlin Wang, D. Burger, K. S. McKinley, S. K. Reinhardt, and C. C. Weems, “Guided Region Prefetching: a Cooperative Hardware/Software Approach,” in _ISCA_ , 2003.
  * [188] P. Zhou, R. Teodorescu, and Y. Zhou, “HARD: Hardware-Assisted Lockset-based Race Detection,” in _HPCA_ , 2007.