# A Comprehensive and Cross-Platform Test Suite
for Memory Safety

Towards an Open Framework for Testing Processor Hardware Supported Security
Extensions

Wei Song<EMAIL_ADDRESS>SKLOIS, Institute of Information Engineering, CAS
School of Cyber Security, University of Chinese Academy of
SciencesBeijingChina ,  Jiameng Ying<EMAIL_ADDRESS>SKLOIS, Institute
of Information Engineering, CAS
School of Cyber Security, University of Chinese Academy of
SciencesBeijingChina ,  Sihao Shen<EMAIL_ADDRESS>SKLOIS, Institute of
Information Engineering, CAS
School of Cyber Security, University of Chinese Academy of
SciencesBeijingChina ,  Boya Li<EMAIL_ADDRESS>SKLOIS, Institute of
Information Engineering, CAS
School of Cyber Security, University of Chinese Academy of SciencesChina ,
Hao Ma<EMAIL_ADDRESS>SKLOIS, Institute of Information Engineering, CAS
School of Cyber Security, University of Chinese Academy of
SciencesBeijingChina and  Peng Liu<EMAIL_ADDRESS>The Pennsylvania State
UniversityUniversity ParkUSA

###### Abstract.

Memory safety remains a critical and widely violated property in reality.
Numerous defense techniques have been proposed and developed but most of them
are not applied or enabled by default in production-ready environment due to
their substantial running cost. The situation might change in the near future
because the hardware supported defenses against these attacks are finally
beginning to be adopted by commercial processors, operating systems and
compilers. We then face a question as there is currently no suitable test
suite to measure the memory safety extensions supported on different
processors. In fact, the issue is not constrained only for memory safety but
all aspect of processor security. All of the existing test suites related to
processor security lack some of the key properties, such as comprehensiveness,
distinguishability and portability.

As an initial step, we propose an expandable test framework for measuring the
processor security and open source a memory safety test suite utilizing this
framework. The framework is deliberately designed to be flexible so it can be
gradually extended to all types of hardware supported security extensions in
processors. The initial test suite for memory safety currently contains 160
test cases covering spatial and temporal safety of memory, memory access
control, pointer integrity and control-flow integrity. Each type of
vulnerabilities and their related defenses have been individually evaluated by
one or more test cases. The test suite has been ported to three different
instruction set architectures (ISAs) and experimented on six different
platforms. We have also utilized the test suite to explore the security
benefits of applying different sets of compiler flags available on the latest
GNU GCC and LLVM compilers.

##  1\. Introduction

It is widely known that the lack of memory safety in low-level languages like
C/C++ enables attackers to maliciously access data in the memory, alter the
value of key variables such as code pointers, hijack the behavior of
applications, or even take full control of a computer system (Szekeres et al.,
2013). In addition to the endless endeavor to discover and then fix the
individual bugs scattered around the enormous software ecosystem, researchers
have proposed a multitude of defense techniques in recent decades to thwart
one or multiple types of attacks exploiting memory bugs. However, memory
safety remains a critical and widely violated property in reality as most of
the operating systems (OSes) and applications running on the off-the-shelf
computers are not adequately defended.

Some of the early defense techniques, such as stack canary (Cowan et al.,
2003b), data execution prevention (DEP) and address space layout randomization
(ASLR) (the PaX Team, 2001), have been widely adopted by commercial
processors, operating systems (OSes) and compilers, which in turn triggers the
evolution of the more complicated and evasive variants of attacks, such as
return-oriented programming (ROP) (Shacham, 2007; Tran et al., 2011), jump-
oriented programming (JOP) (Bletsch et al., 2011), counterfeit object-oriented
programming (COOP) (Schuster et al., 2015) and data-oriented programming (DOP)
(Chen et al., 2005; Hu et al., 2016). To battle with these new attacks,
numerous defense techniques have been proposed and developed. Some of the
extensively studied techniques include type enforcement by tagging the
pointers (Necula et al., 2005; Arora et al., 2006; Kwon et al., 2013; Dhawan
et al., 2014; Woodruff et al., 2014), pointer authentication (PA) (Cowan et
al., 2003a; Mashtizadeh et al., 2015), code-pointer integrity (CPI) (Kuznetsov
et al., 2014), control-flow integrity (CFI) (Abadi et al., 2009; Burow et al.,
2017) and data-flow tracking (Chang et al., 2008; Song et al., 2016). Yet most
of them are not applied or enabled by default in production-ready environment
due to their substantial running cost. For example, CFI has been implemented
in LLVM and GCC (Tice et al., 2014) but it is almost disabled by default. The
support of Intel MPX (memory protection extension) (Ramakesavan et al., 2016)
was added but later dropped by major compilers. As a result, even naive buffer
overflow bugs can be exploited without being detected.

On the bright side, the situation might change in the near future, since the
hardware supported defenses against the more complicated and evasive variants
of attacks are finally beginning to land on commercial processors, OSes and
compilers. The Intel control-flow enforcement technology (CET) (Intel
Corporation, 2019), initially drafted in 2016 to thwart ROP and JOP related
attacks, is included in the 11th generation Tiger Lake processors (expected in
the middle of 2021). It will be supported by Windows 10 through the hardware-
enforced stack protection. The Arm pointer authentication (PA) was added in
the ARMv8.3-A (Brash, 2016) to facilitate defenses like CPI. It has already
been implemented in the Apple A-series processors after A12 and supported in
LLVM from version 8 (ARM, 2021). The Arm memory tagging extension (MTE) was
proposed in ARMv8.5-A (Bannister, 2019) to further enhance memory safety in
general and Google has announced its willingness to support it in Android
(Serebryany and Herle, 2019).

With no doubt, hardware supported security extensions would significantly
reduce the cost of applying defenses and gradually spread to all computer
architectures and processor implementations. This tide of changes would bring
us a new set of questions:

  * •

_Q1: How to compare the security provided by different processors?_ When
picking a proper processor (actually a minimal platform comprising of a
processor, an OS and a compiler) for a specific system to be developed, a
system designer may desire to choose the one providing the maximum security
protection from all candidate processors within the performance and cost
budget.

  * •

_Q2: How to choose the right compiler and compiler flags (OS features) for the
best security-performance trade-off?_ When shipping an application to a target
processor, an application designer might wish to apply a proper set of
compiler flags on a suitable compiler that enables the most security
extensions available on the target processor while maintaining the performance
target.

  * •

_Q3: How to evaluate the security benefit of a new defense on a certain
processor?_ As security researchers, we would also like to systematically
evaluate the security benefit of a newly proposed defense on a target
processor.

We believe the answer is to create a test suite which is comprehensive enough
to quantitively evaluate the safety of a processor while also portable enough
to comparatively evaluate multiple processors and be expanded to evaluate new
defenses, vulnerabilities and architectures. Obviously, this will be an
extremely challenging while long-lasting work.

As an initial step, we propose an expandable test framework for measuring the
processor security and open source a memory safety test suite utilizing this
framework. The framework is deliberately designed to be flexible so it can be
gradually extended to all types of security features supported by the
processor architecture and hardware (micro-architecture), such as defenses
targeting the cache side-channels and speculative execution attacks.

The initial test suite for memory safety currently contains 160 test cases
covering spatial and temporal safety of memory, memory access control, pointer
integrity and control-flow integrity. Each type of vulnerabilities and their
related defenses have been individually evaluated by one or more test cases.
The test suite has been ported to three different instruction set
architectures (ISAs) and experimented on six different platforms. We have also
utilized the test suite to explore the security benefits of applying different
sets of compiler flags available on the latest GNU GCC and LLVM compilers.

The rest of this paper is organized as follows: Section 2 explains the
motivation behind this work. Section 3 describes the assumptions made in this
paper. Section 4 illustrates the testing framework proposed to construct and
run the test suite. The available test cases are summarized in Section 5 with
the implementation challenges illustrated in Section 6. Section 7 analyzes the
test results on the six ported platforms. Section 8 discusses the related and
future work. Finally, the paper is concluded by Section 9.

##  2\. Motivation

This work is motivated by the lack of testing for processor security in both
industry and academia.

###  2.1. The Lack of Testing in Industry

The systematic testing for processor security is almost non-existing in
industry. Major processor manufacturers and OS vendors may have internal tests
but they are not in any way shared or publicized. They normally sit on the
back seat and are passively driven to respond to vulnerabilities only when
they are discovered and even already publicized. Take the discovery of
Meltdown and Spectre for example, it was reported that Intel did not inform
the U.S. National Security Agency until they were made public (Nellis, 2018).
Microsoft eventually mobilized hundreds of people across the company and
urgently brought in external experts in response and negotiated an extension
of the disclosure date to 120 days (Fogh and Ertl, 2018). All of these
indicate that the transient execution vulnerabilities have struck the whole
industry off guard. If processor designs and the manufactured central
processing units (CPUs) were systematically and regularly tested for hardware
security, the situation might be significantly improved, at least the response
would be much more prompt. However, there is no evidence of such testing
facility being proposed or developed. The mitigation of processor
vulnerabilities are still internally evaluated, responded and potentially
resolved by individual processor manufacturers. This would leave the smaller
players, who cannot spare engineers to respond to these security concerns or
simply are not knowledgeable enough to handle these concerns, out in the
insecure wild along with their customers.

The situation on the customer side is even more stark. End customers are
forced to unconditionally trust the security claims from processor
manufacturer without any means to verify them. Even governments may have very
limited ways to measure the security of the processors deployed on the safety
critical infrastructures.

If there is a generic testing framework capable of measuring the processor
security, even if the measurement covers only the publicized hardware
vulnerabilities, processor manufacturers may benefit from it by obtaining an
early warning on the potentially unprotected vulnerabilities and the customer
can earn extra confidence on the security claims produced by the
manufacturers, as well as a mean to compare the security levels of different
CPUs.

###  2.2. Insufficient Testing in Academic Research

The research in academia is normally more advanced and open than those in
industry, which is the same for the security evaluation for processors.
However, the existing research is still insufficient.

Some test suites have already been proposed in the literature. The Juliet
C/C++ and Java test suite (Boland and Black, 2012) is a widely used collection
of flawed programs to test the effectiveness of static analyzers in compilers.
RIPE (runtime intrusion prevention evaluator) (Wilander et al., 2011) provides
a combination of synthetic buffer overflow attacks as a way to test the
effectiveness of related defenses. RIPE is perhaps the most relevant and well-
known test suite for processor security and has already been extensively used
in architectural researches to verify their defenses against control-flow
attacks (Kuznetsov et al., 2014; Song et al., 2016), especially the ROP
variants. Each test case in RIPE is a relatively complete attack specified by
five factors: the location of the buffer to overflow, the type of code pointer
to corrupt, the type of overflow attack, the type of shell code and the
function to be abused in the overflow attack. Since each factor has multiple
choices, RIPE contains as many as 850 test cases to cover the ROP variants of
control flow alteration caused by buffer overflow. It is obvious that this way
of testing (by relatively complete attacks involving multiple attack factors)
and reaching coverage (by exhausted testing) is very hard to be extended into
a full-fledged suite for all aspects related to processor security.

None of the existing test suites can, or be expanded to, conduct a thorough
measurement of the safety of a processor due to the lack of some of the
following key properties:

  * •

_Comprehensiveness_ : The test suite should have a wide coverage of all types
of vulnerabilities and defense techniques especially those potentially
supported by the processor hardware. Most of existing test suites either cover
only a portion of the vulnerabilities and protections, such as RIPE, or
concentrate on only the software defenses, such as Juliet.

  * •

_Distinguishability_ : To provide a quantitative evaluation, the test suite
should provide per-vulnerability and per-defense test cases which concentrate
on a single attack factor (such as one type of target or one attack
technique). Such level of distinguishability is not available in RIPE.

  * •

_Structure_ : The test suite should be produced in a structured way rather
than a collection of ad-hoc test for individual vulnerabilities or defenses.
This structure is crucial for the test suite to reach a coverage which is
understandable and reasonable.

  * •

_Portability_ : The test suite should be able to be ported to multiple
platforms using various computer architectures, OSes and compilers with a
minimum effort. Once ported to a certain type of platforms, the test suite
should run out-of-the-box on all variants of the same platform. When a new
type of attacks or protection emerges, individual test cases should be able to
be added with a minimum effort. None of the existing test suites maintain good
portability.

In this work, we seek to provide these key properties in a expandable testing
framework by the following method.

  * •

_Comprehensiveness_ : It is impossible to bring up a comprehensive test suite
for all aspects of processor security in a short period of time but we can
gradually grow the aspects covered by the test suite. As in the initial test
suite, major aspects of the memory safety are covered.

  * •

_Distinguishability_ : Every test case in the test suite is made concentrating
on a single type of vulnerabilities or defenses. The runtime context of
individual test cases are created by a rather brutal forced way, as shown in
Section 4.3, to deliberately avoiding introducing extra vulnerabilities. To
some extent, each test case is a partial attack that is normally executed as a
single step in a relatively complete attack.

  * •

_Structure_ : The dependence of individual types of vulnerabilities and
defenses are analyzed and recorded using a relation graph, as described in
Section 4.2. By utilizing this graph, test cases can collect crucial pre-
knowledge from the results of other tests and use it to improve test accuracy.
The order of testing and a reasonable coverage are also derived form this
relation graph.

  * •

_Portability_ : The code of individual test cases are divided into platform
independent and dependent parts. All the platform dependent parts are moved to
a shared platform-specific library as described in Section 4.3. This
significantly improves the portability of the test suite.

##  3\. Threat Model

_Adversarial capability_ : We assume unprivileged attackers with the ability
to execute and control the input of user programs on an OS. It is also assumed
that user programs commonly contain memory safety vulnerabilities which might
be exploited to achieve arbitrary reads and writes into the program address
space. As a consequence, any attacks utilizing the existing memory safety
vulnerabilities to leak information, corrupt data, and hijack control flow are
in scope. We limit the current test suite to attacks utilizing the existing
memory safety vulnerabilities to attack the user level data space. Therefore,
attacks that using side-channels, such as cache side-channels (Yarom and
Falkner, 2014; Yan et al., 2017) and transient execution attacks (Canella et
al., 2019a), and attacks targeting the kernel space are out of the scope of
this paper but, of course, they are the targed attacks for the future expended
version of this test suite.

_Definition of a platform_ : The safety of a processor is measured by running
the test suite on a minimum running environment, namely a platform, comprising
the processor under test and a minimum OS running on it. This minimum OS
includes a kernel and a small number of runtime libraries just enough for the
execution of the user programs potentially under attack. All user programs are
assumed to be compiled by a common compiler using similar compiler flags.

_Hardening assumptions_ : We assume some memory safety protection features
have been implemented by either software or hardware but the exact details of
these features may not make available to the test suite unless they are
visible to user-level programs (through environmental variables) or indicated
by compiler flags fed to the test suite. If a safety protection feature is
implemented, it should have already been supported by the OS and the compiler;
therefore, the protection is enabled by default or can be switched on by
compiler flags.

##  4\. The Framework of Testing

This section proposes a flexible testing framework suitable for measuring the
security provided by a processor. While in the description, we use memory
safety as the targeted aspect of security for measurement.

###  4.1. Scope of Testing

Memory safety could be viewed as a set of memory safety properties. All memory
corruption vulnerabilities and exploitation techniques rely on the absence of
certain memory checks on one or more properties which allow values in memory
to be maliciously leaked or altered against their original meaning in the
program.

For example, a buffer overflow attack happens when the action of modifying the
value of a buffer goes beyond its boundary. Modifying beyond the buffer
boundary is the memory safety property that has been violated due to the lack
of a proper boundary check. The fact that the modifying of the buffer can be
made beyond the boundary is a vulnerability. To eliminate this vulnerability,
the specific buffer can be patched with the missing boundary check in the
source code, or this check can be universally added to all buffers to prevent
them from being exploited by the same type of vulnerabilities. We consider the
latter as a defense. A defense can be implemented purely by software but with
significant slowdown (Nagarakatte et al., 2009; Serebryany et al., 2012), or
it can be enforced by hardware through memory tagging or fat pointers while
with the help of compilers and libraries (Nagarakatte et al., 2012). We
consider the former as a pure software defense while the latter as a hardware
supported security enhancement.

Memory safety is not the responsibility of the processor hardware but the
whole system, including the programming language, the compiler, the operating
system, the runtime libraries, the architecture and finally the processor
hardware. Not all memory safety properties are enforceable by the processor
hardware but when they are, adopting the hardware supported enforcement
normally brings significant benefit in performance. The objective of the
proposed test suite is to measure the security boost brought by the security
enforcement potentially implemented in the processor under test. For this
reason, the aforementioned buffer overflow attack should be tested because the
missing boundary check can be enforced by the processor hardware through
memory tagging or fat pointers. In other words, _if the absence of a memory
check can be exploited by attackers and the check can be efficiently enforced
by the processor hardware (and architecture) with (or even without) the help
of software, this check is considered a potential hardware supported security
enhancement that should be covered by the test suite._

Although the concept might be easy to digest, the reality is still very much
complicated. At least for memory safety, most architectural security
enhancements require software (especially the compilers) involvement. The test
suite aims to discover whether certain checks have been enforced with the
support from the processor hardware; however, the fact that a test suite must
be executed in a software environment blurs the boundary between software and
hardware. A test might find the existence of a check but it cannot simply tell
whether it is hardware supported or purely software enforced. As a result,
_the test suite should be considered as a best-effort approach to measure the
memory safety supported by the processor hardware._ The test suite should be
compiled by compilers pre-installed in the production-ready OS and run on the
OS with a minimal execution environment where only the necessary libraries are
installed. In this scenario, checks (defenses) found by the test suite are
likely to be enforced with the support from processor hardware (or at least
the essential runtime library) rather than some third-party software.

With all the above in mind, we consider _the processor hardware, the ISA, the
kernel of the production-ready OS, the default compiler, and the essential
runtime libraries_ as the scope of testing. This leaves the security features
available in some binary-level translation and execution tools, and kernel
patches (such as Grsecurity® (Open Source Security, Inc, 2021)) out of the
scope. Defenses that are usually implemented and enforced purely by software
are not deliberately evaluated while each hardware enforceable memory check
(defense) should be covered by a specific test case. Similarly, the test suite
does not verify the existing software running on the target platform,
including the kernel and the runtime libraries, is free of vulnerabilities. In
fact, it assumes all programs may contain vulnerabilities but they would
become difficult to exploit when proper defenses are enforced.

###  4.2. Definition of Coverage

Existing test suites choose to launch complete attacks exploiting a number of
vulnerabilities and utilizes multiple attack techniques at different stages
(Poe and Li, 2006; Wilander et al., 2011). If a coverage is defined as all
possible ways of attacks, it would need to cover all the potential
combinations of vulnerabilities and attack techniques, which becomes a task
impossible to fulfill. As a result, existing test suites normally cover only a
set of related types of vulnerabilities, leaving a significant amount of
attack variants uncovered (Li et al., 2020).

Instead of trying to reach a partial coverage by launching complete attacks,
this test suite tries to cover a wide range of vulnerabilities with a large
number of small and concentrated test cases. Each test cases endeavors to
check the existence of a single type of memory vulnerabilities or the absence
of a specific memory check. By doing this, the test suite satisfies the
requirement for the first two key properties, _comprehensiveness_ and
_distinguishability_ , as described in Section 2.

Since each test case concentrates on a single type of vulnerabilities, which
is normally exploited as a step in a complex attack, test cases are not
mutually isolated but rather rely on each other to provide key information. We
try to describe this relationship between vulnerabilities by a graph, which
then helps the test suite execute all test cases with a proper order and reach
a coverage faster. More importantly, we can derive a clear definition of
coverage based on this graph. Three types of relations have been considered:

  * •

_Dependency_ : If a type of vulnerabilities _A_ can be exploited only when
another type _B_ is exploitable, _A_ depends on _B_. Take the classic ROP
attack as an example. To alter the return address stored on the stack and make
it point to a code gadget, the attacker needs to know the address of the
gadget, which usually depends on that some code pages are readable as ASLR is
enforced. In this case, _A: alteration of return address_ depends on the _B:
readability of code pages_.

  * •

_Specialization_ : If a constraint is applied on a type of vulnerabilities _B_
, which creates a new type of vulnerabilities _A_ , _A_ is a special case of
_B_. Let us use the same ROP example. Since the alteration of return address
happens on the stack, it can be considered as applying a type constraint (code
pointer) on a general stack modification vulnerability. Therefore, _A:
alteration of a return address_ is a special case of _B: data alteration on
the stack_.

  * •

_Relaxation_ : If two types of vulnerabilities _A_ and _B_ are similar but _B_
is more evasive than _A_ in the view of memory checks, _A_ is a relaxed form
of _B_. This type of relation could be hard to conceive but there are abundant
examples in practice. Still considering the ROP attack, replacing the return
address with a wrong call site address is a more evasive attack than replacing
it with a non-call-site address. In this case, _A: ROP using a non-call-site
address_ is a relaxed form of _B: ROP using a wrong call site address._

In all types of relation, if _B_ is found eliminated by a memory check, _A_ is
almost certainly eliminated by the same memory check and is therefore
unnecessary to be tested. In this scenario, we call _B_ the prerequisite of
_A_.

Figure 1. Relation graph related to ROP attacks

The prerequisites of all test cases can be depicted by a labeled directed
graph, which is called a relation graph in this paper. The one describing the
ROP related vulnerabilities is shown in Figure 1. Each node denotes a test
case targeting one specific type of vulnerabilities or defenses while an arc
between two nodes describes the prerequisite relation. The three types of
relations are depicted in different colors and dash lines. To hijack a return
address, the attacker needs to search for a proper gadget in the code section.
This could be done by either using an offset collected from static binary
analysis when ASLR is not enabled or analyzing code pages at runtime (Snow et
al., 2013). Therefore, ROP (call site) _depends_ on the missing of ASLR or the
possibility to read code page at runtime. Since the return address is a
special variable stored on the stack, altering it is a _special case_ of stack
overflow attacks. Consequently, ROP (call site) takes four over/underflow
tests as prerequisites as well. Three ROP tests are included in the relation
graph: returning to a call site, using a function as a gadget, and returning
to any code. Since ROP (call site) is the most evasive of the three, we
consider ROP (function) and ROP (any code) as two _relaxed_ cases. Finally,
the gadgets could be codes directly injected in read-only data, global data,
heap and stack. Depending on the location, we have four _special cases_ of ROP
(any code): code injection in rodata, data, heap and stack.

The prerequisites for ROP (call site) is complicated. It has six prerequisites
but it is enabled when _one of_ ASLR and read code page passes, and _one of_
the four over/underflow tests passes. This is literally a two-level
relationship: Prerequisites are divided into subsets. In each subset, the
prerequisite is met if _one of_ the test cases passes. The final prerequisite
is met when the results of all subsets are _AND_ ed together and turns to be
true. To illustrate such a two-level relationship in the relation graph, the
arcs of each subset are labeled by a unique letter. Labels are omitted when
only one level of relation is needed.

According to the prerequisites defined in the relation graph, the test suite
can avoid testing both ROP (function) and ROP (any code) if ROP (call site)
fails. The same is for the four code injection tests if ROP (any code) fails.
_Assuming a relation graph has exhausted all types of vulnerabilities and
defenses related to certain aspects of memory safety, reaching a full coverage
is equivalent to obtain a reachability graph by starting from all nodes
without input arc._ To this end, the test suite satisfies the _structure_
requirement described in Section 2.

We believe the concepts of relation graph and the derived coverage applies to
all aspects of processor security beyond just the memory safety. For example,
a conflict-based cache side-channel attack(Yan et al., 2017) is a multi-step
attack comprising at least the construction of eviction sets, priming the
target cache sets, triggering the victim rerun and finally probing the cache.
The success of each step may depend on some pre-knowledge of the micro-
architecture and certain ways of utilizing the micro-architectural features.
Hardware supported defenses in processors may thwart some of these steps, such
as increasing the difficulties in constructing eviction sets using randomized
caches (Qureshi, 2019; Werner et al., 2019) or effectively prohibiting priming
by enforcing cache partitioning (Liu et al., 2016). All of these attacking
steps, defenses, and micro-architectural features (potential vulnerabilities)
can be described using the relation graph and tested accordingly using a set
of concentrated test cases. In other words, the testing framework can be
expanded to all aspects of processor security.

Another important issue related to relation graph is its construction and
correctness. Currently the relation graph is hand crafted by meanually
building up the dependence between different vulnerabilities, defenses and
related techniques according to our own understanding of the related
literature. Unavoidably this hand crafted relation graph contains human errors
due to our limited knowledge and potential misundertanding. However, the test
suite is opensourced (GitHub repo is provided at the end of this paper) and
the relation graph is recorded in a JSON file (as described by the next
paragraph) included in the test suite. We will gradually evlove the relation
graph along with the expansion of the test suite and our improved
understanding of the related issues. We also hope the test suite would become
a community project which attracts contribution from the whole community
including the corrections on the relation graph.

To utilize the prerequisites at runtime, the relation graph is stored in a
configuration JSON file and analyzed at runtime by a simple test scheduler
written in python. Listing 1 demonstrates a part of the configuration file
describing Figure 1. Every record is a test case depicted as a node in the
relation graph, with the name mapping between JSON records and nodes in the
relation graph described in Table 1. The prerequisites are listed in the
“require” property, which is a list of lists. Each inner list contains
prerequisites within the same subset while results of all inner lists of are
_AND_ ed together to decide whether the case is tested.

⬇

1{

2 ... // omit some code

3

4 "return-to-wrong-call-site": {

5 "require": [ [ "read-func", check-ASLR"],

6 [ "overflow-write-index-stack",

7 "underflow-write-index-stack",

8 "overflow-write-ptr-stack",

9 "underflow-write-ptr-stack" ] ]

10 },

11

12 "return-to-non-call-site": {

13 "require": [ [ "return-to-wrong-call-site" ] ]

14 },

15

16 "return-to-func": {

17 "require": [ [ "return-to-wrong-call-site" ] ]

18 },

19

20 "return-to-instruction-in-rodata": {

21 "require": [ [ "return-to-non-call-site" ] ]

22 },

23

24 "return-to-instruction-in-data": {

25 "require": [ [ "return-to-non-call-site" ] ]

26 },

27

28 "return-to-instruction-in-heap": {

29 "require": [ [ "return-to-non-call-site" ] ]

30 },

31

32 "return-to-instruction-in-stack": {

33 "require": [ [ "return-to-non-call-site" ] ]

34 },

35

36 ... // omit some code

37}

Listing 1: configure.json Table 1. Name mapping between JASON and Figure 1 Node Name | JSON record name
---|---
read code page | read-func
ASLR | check-ASLR
overflow (index) | overflow-write-index-stack
underflow (index) | underflow-write-index-stack
overflow (pointer) | overflow-write-ptr-stack
underflow (pointer) | underflow-write-ptr-stack
ROP (call site) | return-to-wrong-call-site
ROP (any code) | return-to-non-call-site
ROP (function) | return-to-func
code injection (rodata) | return-to-instruction-in-rodata
code injection (data) | return-to-instruction-in-data
code injection (heap) | return-to-instruction-in-heap
code injection (stack) | return-to-instruction-in-stack

###  4.3. Construction of Test Cases

_Portability_ is one of the major concerns in the construction of test cases.
There is no doubt that new types of vulnerabilities will be discovered in the
future, along with new defense techniques. The test suite tries to cover all
the known types of vulnerabilities while allowing new test cases to be easily
added. The ease by which new test cases can be added is then crucial for the
usefulness of the test suite. In addition, new computer architectures and ISA
extensions will be constantly proposed. Instead of painstakingly re-implement
all test cases whenever a new ISA is added or a new OS is supported, the test
suite should try to reuse the platform-independent part while limiting the
platform-specific part to relatively small code segments. For these reasons,
the test suite is composed of two parts: _individual platform-independent test
cases_ and _a shared platform-specific support library_.

Each test case is written in a short (and usually malformed) C++ program
trying to test a single type of vulnerabilities. If the vulnerabilities exist,
the test case should exploit the vulnerability and return zero; otherwise, a
non-zero exit code is used to indicate when and where the test fails or
crashes. Whenever possible, the related vulnerabilities involved in each test
cases is minimized by directly creating a vulnerable context at runtime. Most
code is written in a portable and compiler-independent manner. However, when a
(usually malicious) behavior might be detected and disturbed by a compiler,
extra code might be added to prevent the compiler optimization and snippets of
assembly code might be used to replace the C++ code as most compilers would
not analyze the embedded assembly, let alone optimize it. Since the added
assembly code snippets are unavoidably platform-specific, they are extracted
from test cases and moved to the shared support library.

For example, Listing 2 illustrates the test for return-to-wrong-call-site
which hijacks the return address to a valid call site within static analysis.
The main() function is a bare minimum. It first reads offset from input as it
is later needed by helper(). The value of this offset is actually selected by
the test scheduler, which will be explained in details in Section 6.1. The
main() function then calls helper() twice (line 17 and 23) with the latter one
trying to alter its own return address. If the helper() function succeeds, it
would return to line 19 labeled by main_mid,111 An assembly label is used
because C++ labels are invisible outside the function body.  and exit with a
0. Otherwise, the program may exit with a non-zero code 2, indicating the test
fails to override the return address. The exit code is initialized to 0 and
stored in a volatile global variable grv to escape compiler optimization.

The helper() function is also straightforward. FORCE_NOINLINE stops the
function from being inlined using a compiler attribute. ENFORCE_NON_LEAF_FUNC
inserts some code to guarantee that helper() is not a leaf function and the
return address is always pushed on the stack even on RISC (reduced instruction
set computer) machines. The embedded assembly MOD_STACK_LABEL(main_mid,
offset) replaces the return address with the address of main_mid when the
function is called the second time (grv == 2). The location of the return
address on the stack is pinpointed by the program input offset, which is the
global parameter collected by the main() function indicating the offset to the
stack pointer. As the value of offset varies on different platforms or even
with different compiler flags. This is why it is detected by other tests and
then provided by the test scheduler (see Section 6.1 for more details).
Finally, helper() revises the exit code to 0 just before return. If the return
address is successfully modified and helper() indeed returns to main_mid, the
program would exit as normal (code 0) in the main() function.222 We use exit()
(from cstdlib) here as the call stack is corrupted.

⬇

1#include <cstdlib>

2#include "include/assembly.hpp"

3

4static volatile int grv = 0;

5int offset; // offset to stack pointer

6

7void FORCE_NOINLINE helper() {

8 ENFORCE_NON_LEAF_FUNC;

9 if(++grv == 2) {

10 MOD_STACK_LABEL(main_mid, offset);

11 grv = 0;

12 }

13}

14

15int main(int argc, char* argv[]) {

16 offset = 8 * (argv[1][0] - ’0’);

17 helper();

18

19 DECL_LABEL(main_mid);

20 if(grv == 0)

21 exit(grv);

22

23 helper(); // hijack return address

24 exit(grv);

25}

Listing 2: return-to-wrong-call-site.cpp

Most platform-specific codes are written in embedded assembly and referenced
by macros, which are defined in the shared support library. As shown in
Listing 3, the header file included in test cases (include/assembly.hpp) is
just a wrapper pointing to the correct header prepared for the target
platform. Assuming the target platform is a x86-64 machine, macro __x86_64 is
predefined by most compilers and x86_64/assembly.hpp is then included by
detecting this predefined macro.

⬇

1... // omit some code

2

3#ifdef __x86_64

4 #include "x86_64/assembly.hpp"

5#endif

6

7... // omit some code

Listing 3: include/assembly.hpp

Listing 4 reveals parts of support library for the x86-64 architecture
(x86_64/assembly.hpp). FORCE_NOINLINE is short for a builtin attribute for
non-inline functions. To enforce the non-leaf condition, ENFORCE_NON_LEAF_FUNC
inserts an irremovable external call. MOD_STACK_LABEL(label, offset) is used
to replace the return address stored at [$rsp+offset] with the address of
label in a rather brutal forced manner. This avoids the use of traditional
buffer overflow attacks and probably escapes from most compile-time detection,
and allows the test case to concentrate on the modification of return address
to a non-call-site vulnerability itself.

⬇

1... // omit some code

2

3#define FORCE_NOINLINE __attribute__((noinline))

4

5extern int dummy_leaf_rv;

6extern int dummy_leaf_func(int);

7#define ENFORCE_NON_LEAF_FUNC \

8 dummy_leaf_rv = dummy_leaf_func(dummy_leaf_rv);

9

10// declare a label in assembly

11#define DECL_LABEL(label) asm volatile(#label ":")

12

13// modify stack

14#define MOD_STACK_LABEL(label, offset) \

15 asm volatile( \

16 "lea " #label "(%%rip), %%rax;" \

17 "movl %0, %%ecx;" \

18 "movslq %%ecx, %%rcx;" \

19 "addq %%rsp, %%rcx;" \

20 "movq %%rax, (%%rcx);" \

21 : : "r"(offset) : "rax", "rcx" )

22

23... // omit some code

Listing 4: x86_64/assembly.hpp

Assuming the functions of the macros defined in the shared support library are
modular enough, supporting a new platform should be an easy task. A new
“assembly.hpp” header file should be created to define all the platform-
specific macros and functions using the new architecture. Currently there are
only around 20 macros that are needed to be ported for each new architecture.
This new header file is then added to the common header file
include/assembly.hpp while checked by a predefined macro unique to the new
platform. Adding a new test case for a newly discovered type of
vulnerabilities may involve three steps: (1) Implement the test case using the
shared support library. (2) If new macros are needed, they should be added to
all supported platforms. (3) A record is added in the configuration file.

##  5\. Summary of Test Cases

This section summerize the individual test cases provided by the initial test
suite for memory safety.

###  5.1. Spatial Safety

Spatial safety refers to the property that memory accesses are always in
compliance with the proper data boundaries and the scope of visibility defined
by the program. Any access outside the boundary or the scope of visibility is
considered insecure. It is violated by classic buffer overflow attacks
(Spafford, 1989), which is then used to smash the stack and alter the stored
return address (Aleph One, 1996). Heap attacks also resort to buffer overflow
to access/alter data across the object boundary (Conover and w00w00 Security
Team, 1999). Currently, the test suite has 98 test cases to measure spatial
safety.

####  5.1.1. General read out-of-boundary

16 test cases in total.
overflow-read-index-stack overflow-read-index-heap overflow-read-index-data
overflow-read-index-rodata overflow-read-ptr-stack overflow-read-ptr-heap
overflow-read-ptr-data overflow-read-ptr-rodata underflow-read-index-stack
underflow-read-index-heap underflow-read-index-data underflow-read-index-
rodata underflow-read-ptr-stack underflow-read-ptr-heap underflow-read-ptr-
data underflow-read-ptr-rodata

Both buffer overflow and underflow reads have been tested. We consider two
potential ways to access data out-of-boundary: One is to access with a valid
buffer pointer but an out-of-boundary offset (index) and the other one is to
directly make a previously valid buffer pointer point to an out-of-boundary
location (ptr). For defenses using memory tags (Qin et al., 2005) or fat
pointers (Devietti et al., 2008; Nagarakatte et al., 2009; Kwon et al., 2013),
these two ways may result in exceptions being triggered at different locations
depending on whether the pointer (Cowan et al., 2003a) or the buffer memory
(Qin et al., 2005) is tagged. Since the defense mechanisms on stack, heap,
global data and read-only data sections might be different, they are tested
separately. These tests are the most general cases for out-of-boundary reads
that read only the neighbors of the buffer (overflow by one) without going too
far.

####  5.1.2. General write out-of-boundary

12 test cases in total.
overflow-write-index-stack overflow-write-index-heap overflow-write-index-data
overflow-write-ptr-stack overflow-write-ptr-heap overflow-write-ptr-data
underflow-write-index-stack underflow-write-index-heap underflow-write-index-
data underflow-write-ptr-stack underflow-write-ptr-heap underflow-write-ptr-
data

Similar to out-of-boundary read tests, these tests try to write out-of-
boundary. Both buffer overflow and underflow writes have been tested. Both
index and pointer ways of accesses have been considered. Buffers on stack,
heap and global data sections have been tested separately.

####  5.1.3. Access cross the object boundary

20 test cases in total.
read-cross-object-index-stack read-cross-object-index-heap read-cross-object-
index-data read-cross-object-index-rodata read-cross-object-ptr-stack read-
cross-object-ptr-heap read-cross-object-ptr-data read-cross-object-ptr-rodata
overflow-write-cross-object-index-stack overflow-write-cross-object-index-heap
overflow-write-cross-object-index-data overflow-write-cross-object-ptr-stack
overflow-write-cross-object-ptr-heap overflow-write-cross-object-ptr-data
underflow-write-cross-object-index-stack underflow-write-cross-object-index-
heap underflow-write-cross-object-index-data underflow-write-cross-object-ptr-
stack underflow-write-cross-object-ptr-heap underflow-write-cross-object-ptr-
data

All tests are special cases of the _general read out-of-boundary_ and the
_general write out-of-boundary_ cases. Some defenses may choose to put invalid
data between objects as a tripwire or enforce the boundary check at the
granularity of objects (Devietti et al., 2008). In these cases, the general
overflow vulnerabilities exist but the more severe form of them, accessing
crossing the object boundary, is disabled. To check these types of
vulnerabilities, the test suite tests both read and write accesses, both index
and pointer ways of accessing, and objects on stack, heap, global data and
read-only data.

####  5.1.4. Access cross the stack frame

Four test cases in total.
read-cross-frame-index read-cross-frame-ptr write-cross-frame-index write-
cross-frame-ptr

All tests are special cases of the _general read out-of-boundary_ and the
_general write out-of-boundary_ cases. To thwart overflow attacks against
variables over the frame boundary, some defenses choose to put padding bytes
between the frames (Bhatkar and DuVarney, 2005) while some other defenses use
fat-pointers to enforce data integrity at the frame granularity (Das et al.,
2019). Data compartmentalization at the frame granularity is also effective
(Nyman et al., 2019). The test suite tests both read and write accesses across
the frame boundary using indices or pointers.

####  5.1.5. Access cross pages

Four test cases in total.
read-cross-page-index read-cross-page-ptr write-cross-page-index write-cross-
page-ptr

All tests are special cases of the _access cross the stack frame_ cases. Page
table attributes have long been used to enforce access permissions. System
software or memory allocation library may use the page accessing property to
partially thwart overflow attacks. Since pages are normally larger than stack
frames, we consider these as special cases of _access cross the stack frame_.
The test suite tests both read and write access cross the page boundary using
indices or pointers.

####  5.1.6. Access cross sections

36 test cases in total.
read-cross-section-stack-to-heap-index read-cross-section-stack-to-data-index
read-cross-section-stack-to-rodata-index read-cross-section-heap-to-stack-
index read-cross-section-heap-to-data-index read-cross-section-heap-to-rodata-
index read-cross-section-data-to-stack-index read-cross-section-data-to-heap-
index read-cross-section-data-to-rodata-index read-cross-section-rodata-to-
stack-index read-cross-section-rodata-to-heap-index read-cross-section-rodata-
to-data-index read-cross-section-stack-to-heap-ptr read-cross-section-stack-
to-data-ptr read-cross-section-stack-to-rodata-ptr read-cross-section-heap-to-
stack-ptr read-cross-section-heap-to-data-ptr read-cross-section-heap-to-
rodata-ptr read-cross-section-data-to-stack-ptr read-cross-section-data-to-
heap-ptr read-cross-section-data-to-rodata-ptr read-cross-section-rodata-to-
stack-ptr read-cross-section-rodata-to-heap-ptr read-cross-section-rodata-to-
data-ptr write-cross-section-stack-to-heap-index write-cross-section-stack-to-
data-index write-cross-section-heap-to-stack-index write-cross-section-heap-
to-data-index write-cross-section-data-to-stack-index write-cross-section-
data-to-heap-index write-cross-section-stack-to-heap-ptr write-cross-section-
stack-to-data-ptr write-cross-section-heap-to-stack-ptr write-cross-section-
heap-to-data-ptr write-cross-section-data-to-stack-ptr write-cross-section-
data-to-heap-ptr

All tests are special cases of the _general read out-of-boundary_ and the
_general write out-of-boundary_ cases. Attackers can manipulate the stack data
using a heap pointer (Conti et al., 2015) and probably vice versa. To test
these special attack variants, the test suite checks all possible scenarios
for accessing cross section boundaries.

####  5.1.7. Spray attack

Six test cases in total.
spray-cross-object-stack spray-cross-object-heap spray-cross-object-data
spray-cross-frame spray-cross-page-stack spray-cross-page-heap

All tests are special cases of the _general write out-of-boundary_ cases. In
the access tests, out-of-boundary data are directly accessed using indices or
pointers and leave all the data in between untouched. While in spray tests,
all the data between the buffer to the out-of-boundary target are rewritten.
Such behavior is more likely to be detected by memory padding (Bhatkar and
DuVarney, 2005; Devietti et al., 2008) and tagging (Venkataramani et al.,
2007) than a direct out-of-boundary access. The test suite currently tests six
spray attacks: spray cross the object boundary on stack, heap and global data,
spray cross stack frames, and spray cross pages on stack and heap.

###  5.2. Temporal Safety

Temporal safety refers to the property that data accessed in memory happens
only within the lifetime of the data. Any access before (uninitialized) or
after (use-after-free, UAF) the lifetime is considered insecure. These types
of vulnerabilities potentially lead to corrupted values in memory (Chen et
al., 2005), return-to-libc (Pincus and Baker, 2004), and arbitrary memory read
and write eventually.

Since the target of this test suite is to measure the safety supported by the
processor hardware as described in Section 4.1, its test cases concentrate on
whether a general out-of-lifetime access can happen and consider attacks
specialized to certain memory allocation algorithms (Xie et al., 2016) out of
the scope.

####  5.2.1. Access after free on heap

Three test cases in total.
read-after-free-org-heap read-after-free-alias-heap write-after-free-heap

After an object is released, all pointers pointing to this object become
dangling pointers and should not be used afterward. However, attackers may
utilize these dangling pointers to read the previously released object,
leading to information leakage. Even worse, attackers may write to the just
released memory which potentially causes data corruption. To thwart these
threats, defenses can deliberately nullify all dangling pointers (Lee et al.,
2015; Younan, 2015) or checking whether a pointer is dangling at the
dereference point (Nagarakatte et al., 2012, 2014). The three tests here exam
the possibility of reading/writing a heap object after it is released. For
reading the released object, both the original pointer or a copy of it (alias)
have been tested as a way to measure the completeness of pointer
nullification.

####  5.2.2. Reclaim on heap

Only one test case.
reallocate-heap

Once a dangling pointer falls into the control of attackers, it can be used to
launch targeted data corruption if the released memory is reallocated to the
same type of objects. Some secure memory allocators try to prevent objects
from reclaiming the previously released memory (Novark and Berger, 2010;
Akritidis, 2010) but attackers may force the memory allocator to do so (Lee et
al., 2015). To measure this vulnerability, this test tries to force the memory
allocator to place an object with the same type at the same location where it
is previously released.

####  5.2.3. Access after reclaim on heap

Three test cases in total.
access-after-reclaim-heap write-before-reclaim-heap write-after-reclaim-heap

All tests depend on the _reclaim on heap_ test. Assuming attackers have
successfully forced an object to reclaim the same memory space previously
released, they would then try to manipulate the object using a dangling
pointer. access-after-reclaim-heap tests whether an uninitiated variable
belonging to the newly allocated object retains its previous value. In case
the memory is safely cleaned when it is released, write-before-reclaim-heap
tries to maliciously initiate this uninitiated variable just before the memory
is reclaimed. Finally, write-after-reclaim-heap tests whether a dangling
pointer can be used to corrupt an object after it reclaims the same memory
space.

####  5.2.4. Access after free on stack

Two test cases in total.
read-after-free-alias-stack write-after-free-stack

UAF attacks happen on stack as well. When a function returns, the stack frame
is released for the next function call. If a dangling pointer pointing to the
released stack frame is mistakenly leaked by a function argument, a global
variable or even the return value, it allows attackers to manipulate the frame
when it is reclaimed (Younan, 2015). Similar to the tests for UAF on heap, the
tests start with reading a released stack variable to see if it retains its
value and then write it to see whether the released frame space is protected.

####  5.2.5. Reclaim on stack

Only one test case.
reallocate-stack

The memory structure of the stack frame for the same function is normally the
same every time the function is called. This potentially allows attackers to
precisely manipulate the variables on the stack using a dangling pointer.
Fine-grained runtime stack layout randomization (Aga and Austin, 2019) may
stop such attacks by locating stack variables at different positions whenever
a function is called again. This test verifies whether the stack layout is
changed when a function is called again.

####  5.2.6. Access after reclaim on stack

Three test cases in total.
access-after-reclaim-stack write-before-reclaim-stack write-after-reclaim-
stack

All tests are dependent on the _reclaim on stack_ test. Assuming the stack
layout remains the same for the same function, access-after-reclaim-stack
tests whether an uninitiated variable keeps its value from the previous
function call, write-before-reclaim-stack tries to maliciously initiate the
uninitiated variable before the next function call. Finally, write-after-
reclaim-stack tests whether a dangling pointer can be used to alter a stack
variable when the stack frame is reclaimed by a new function call.

###  5.3. Access Control

We consider all defense techniques that exert restrictions on accesses as
_access control_ related defenses. ASLR (Forrest et al., 1997) normally acts
as the first level of defense against code-reuse attacks. It prohibits
attackers from collecting the addresses of the required gadgets by static
binary analysis. DEP is another widely adopted defense that disallows codes in
writable pages from being executed. To thwart the dynamic code-reuse attacks
which collect gadgets at runtime (Snow et al., 2013), both code randomization
(Sinha et al., 2017) and read exclusive execution (R$\oplus$X) (Pomonis et
al., 2019) have been proposed to forbid attacker from reading executable
pages.

####  5.3.1. Access control

Three test cases in total.
check-ASLR read-func read-GOT

check-ASLR checks whether the address space is randomized, read-func checks
whether the body of a function (in a code page) can be read, and finally read-
GOT checks whether an entry in the global offset table (GOT) can be read.

###  5.4. Pointer Integrity

We consider all attacks that directly cause a malicious alteration of control
flow as control-flow related attacks and treat all defense techniques against
such attacks as control flow related defenses. Most defenses fall into two
categories: _pointer integrity_ and _control-flow integrity_. The former
prevents pointers (mostly code related) from being maliciously modified
(Kuznetsov et al., 2014) while the latter tries to stop those altered pointers
from affecting the control flow (Abadi et al., 2009).

Since most control-flow attacks rely on the alteration of certain key
pointers, the tests for pointer integrity are considered as prerequisites for
the tests for control-flow integrity. This set of tests aim to cover the
alteration of different types of key pointers. Note that the alteration of
return addresses is closely related to ROP attacks and is thus tested there.

####  5.4.1. Function pointer

Two test cases in total.
func-pointer-assign func-pointer-arithmetic

Function pointers are one of the most frequently attacked targets. func-
pointer-assign checks whether a function pointer can be altered with embedded
assembly (therefore bypassing the analysis and potential protections from
compilers through pointer authentication (Mashtizadeh et al., 2015) or tagging
(Kuznetsov et al., 2014; Song et al., 2016)). It is very rare that arithmetic
operations occur on function pointers (Chen and Chisnall, 2019). Function
pointers are normally cloneable but not mutable. By applying unnecessary
arithmetic operations on a function pointer, func-pointer-arithmetic tests
whether such behavior raises any exceptions.

####  5.4.2. VTable pointer

Two test cases in total.
read-vtable-pointer write-vtable-pointer

Virtual table (VTable) pointers are the major targets in COOP attacks
(Schuster et al., 2015). Defenses like CPI can protect VTable pointers just
like function pointers (Kuznetsov et al., 2014) and prevent them from being
modified. This protection is tested by write-vtable-pointer. Since VTable
pointers should not be explicitly accessed by any source-level program, and
COOP attacks usually need to read and reuse VTable pointers, read-vtable-
pointer checks whether deliberately reading a VTable pointer would trigger any
exceptions.

####  5.4.3. Global offset table

Only one test case.
modify-GOT

GOT is used by dynamically linked programs to search for (function and data)
symbols defined in shared libraries at runtime. Since the content of the GOT
is updated at runtime, GOT might be stored on a writable page. This
potentially allows attackers to hijack library functions by altering the
corresponding GOT entries (c0ntex, 2006). The provided test modify-GOT checks
whether an entry in the GOT can be hijacked to an attacker controlled
function.

###  5.5. Control-Flow Integrity

Control flow refers to the indirect jumps caused by function returns, function
calls and other compiler added jumps. By altering the return address stored on
the stack, attackers can redirect the control flow to code chosen or injected
by them, which is the basis for ROP attacks (Shacham, 2007). Similarly,
maliciously affecting the pointers used in these indirect calls and jumps
(Bletsch et al., 2011; Schuster et al., 2015) can hijack the control flow as
well. Currently, the test suite has 41 test cases to measure the backward
control-flow integrity:

####  5.5.1. Return to injected code

Four test cases in toal.
return-to-instruction-in-stack return-to-instruction-in-heap return-to-
instruction-in-data return-to-instruction-in-rodata

All tests are special cases for _return-to-non-call-site_. Although attacks
relying on direct code injection are largely thwarted by DEP, the same defense
may not be available on legacy systems and embedded systems. As a result, this
test suite still checks whether a function can be hijacked and returned to an
injected code. Note that injecting code using constant texts (read-only data)
remains viable on most systems unless some forms of R$\oplus$X is enforced
(Pomonis et al., 2019).

####  5.5.2. Return to existing code

Six test cases in total.
return-to-wrong-call-site-within-static-analysis return-to-wrong-call-site
return-to-non-call-site return-to-func return-to-libc return-without-call

All tests are relaxed cases of return-to-wrong-call-site-within-static-
analysis. return-to-non-call-site represents the common ROP attacks that use
arbitrary code snippets as gadgets. To avoid the detection of coarse-grained
CFI checks, attackers may be forced to use call preceded gadgets (return-to-
wrong-call-site) (Carlini and Wagner, 2014), long and benign code segment
(return-to-func (Göktas et al., 2014) and return-to-libc (Pincus and Baker,
2004)), or even previously returned locations (replay attack, return-to-wrong-
call-site-within-static-analysis (Roessler and DeHon, 2018)). When multiple
gadgets are chained together, the call-return pair becomes unbalanced, which
should be easy to detect by a shadow stack (Dang et al., 2015; Burow et al.,
2019). This type of unbalance is tested by return-without-call.

####  5.5.3. Call to injected code

Four test cases in total.
call-instruction-in-stack call-instruction-in-heap call-instruction-in-data
call-instruction-in-rodata

All tests are special cases of call-mid-func. Similar to the tests for
backward control-flow integrity, these tests check whether a function call can
be hijacked to a code injected by attackers.

####  5.5.4. Call to existing code

Three test cases in total.
call-wrong-func-within-static-analysis call-wrong-func call-mid-func

All tests are special cases of call-wrong-func-within-static-analysis. call-
mid-func is the most general case as the function pointer is hijacked to an
arbitrary location. Such attacks can be easily detected by most CFI checks
(Abadi et al., 2009; Burow et al., 2017) as the arbitrary location is rarely a
function entry point. To avoid detection by coarse-grained CFI checks (Davi et
al., 2014), call-wrong-func hijacks a function pointer to another function
while the function chosen by call-wrong-func-within-static-analysis also falls
in the valid set of targets collected by static control-flow analyses (thus is
detectable only to path-sensitive CFI defenses (Ding et al., 2017; Zhang et
al., 2018)).

####  5.5.5. Call with wrong arguments

Nine test cases in total.
call-wrong-num-arg-func call-wrong-type-arg-int2double-func call-wrong-type-
arg-op2doublep-func call-wrong-type-arg-fp2dp-func call-wrong-type-arg-dp2fp-
func-stack call-wrong-type-arg-dp2fp-func-heap call-wrong-type-arg-dp2fp-func-
data call-wrong-type-arg-dp2fp-func-rodata

All the call-wrong-type-arg-dp2fp-func-xxxx test cases depend on the call-
instruction-in-xxxx. Some research claims that even the fine-grained CFI
defenses can be circumvented by corrupting only the arguments of functions
(Evans et al., 2015) while the types of arguments can be used to enhance the
accuracy in detecting COOP attacks (van der Veen et al., 2016). To cover these
attack and defense variants, call-wrong-num-arg-func check whether a function
can be called with a mismatched number of arguments. Extra tests are provided
for function calls with mismatched types of arguments, such as providing an
integer for a double argument (int2double), providing an object pointer for a
double or an integer pointer (op2doublep and op2intp), and providing a
function pointer to a data pointer (fp2dp). The last four tests check whether
injected code can be called by placing a data pointer in the place for a
function pointer (dp2fp).

####  5.5.6. VTable injection

Three test cases in total.
call-wrong-func-vtable-stack call-wrong-func-vtable-heap call-wrong-func-
vtable-data

Modern compilers put VTables on read-only pages to prevent them from being
altered. However, attackers can forge fake VTables in other writable spaces
and using them to dislodge the real ones (Schuster et al., 2015). These three
tests check whether VTables can be replaced with a fake one created on stack,
heap and global data.

####  5.5.7. Replace VTables with existing tables

Seven test cases in total.
call-wrong-func-vtable-parent call-wrong-func-vtable-child call-wrong-func-
vtable-sibling call-wrong-func-vtable call-wrong-func-vtable-released call-
wrong-func-vtable-offset call-wrong-num-arg-vtable

Assuming VTables cannot be forged, an attacker would try replacing them with
existing ones inside the memory. To thwart this type of COOP attacks, class
hierarchy analysis (Jang et al., 2014) tries to limit the tables that are
available to attackers. call-wrong-func-vtable checks the most general case
where a VTable is replaced by an arbitrary table existing in memory. The test
suite then checks whether the VTables from a parent class, a child class or a
sibling class can be used for replacement. call-wrong-func-vtable-released
tests whether a VTable pointer of a released object can be reused in a live
object (Sarbinowski et al., 2016). call-wrong-func-vtable-offset checks
whether a VTable pointer can be added with a small offset. Finally, call-
wrong-num-arg-vtable checks whether the virtual function can be replaced with
another one with different number arguments; therefore, allowing attackers to
manipluate the arguments.

####  5.5.8. Indirect jump

Five test cases in total.
jump-instruction-in-stack jump-instruction-in-heap jump-instruction-in-data
jump-instruction-in-rodata jump-mid-func

jump-mid-func checks whether an arbitrary code snippet existing in memory can
be used by a general JOP attack. All jump-instruction-in-xxxx test cases are
special cases of jump-mid-func. Similar to the tests for ROP and COOP attacks,
these four tests check whether a JOP attack (Bletsch et al., 2011) can hijack
the jump target to an injected code.

##  6\. Challenges in Implementation

In this section, we describe the challenges encountered during the
implementation and our solutions to resolve them.

###  6.1. Obtain Parameters at Runtime

Some test parameters are only reliably available at runtime. One such example
is the offset of the return address on the stack from the stack pointer, as
mentioned in Section 4.3. The value of offset depends on the ABI (application
binary interface) definition of the target architecture, whether the frame
pointer is pushed on the stack, and whether code instrument affects the stack,
such as initializing the canary for stack smashing protection (Cowan et al.,
2003b). The value of this offset is therefore hardly fixed but all ROP related
test cases rely on this parameter.

In this test suite, the value of offset is obtained from a blind testing using
return-to-wrong-call-site, which is the common prerequisite for all ROP tests,
and fed to all depended tests as an input argument. As shown in Listing 2 and
described in Section 4.3, the test gets an offset from input and checks
whether the return address can be hijacked to a seemly valid return site
main_mid. Since this offset is unknown, the test suite tries to get the
correct value by launching the same test multiple times with different values
of offset. Such behavior is defined in the configuration file through an
extensions illustrated in Listing 5.

⬇

1{

2 ... // omit some code

3

4 "return-to-wrong-call-site": {

5 ... // omit some code

6 "arguments": ["-roffset"],

7 "offset": [0, 8, 1],

8 "set-var": "stack-offset"

9 },

10

11 "return-to-non-call-site": {

12 "require": [ [ "return-to-wrong-call-site" ] ],

13 "arguments": ["-vstack-offset"]

14 },

15

16 ... // omit some code

17}

Listing 5: Feed runtime parameter between test cases

The list recorded in property arguments (line 6 in Listing 5) defines the
input arguments for a test. Arguments are simply attached to the test case
except for the special ones starting with “-”. These special arguments always
take the form of “-$t$name”, where $t$ denotes the type and name identifies
the property defining this argument. Currently, we support three types: range
(‘r’), list (‘l) and variable (‘v’). When an argument takes the form of
“-$r$name”, it actually defines a range (“-r”) defined in a property named
“name”. In the case of return-to-wrong-call-site, a range of [0:7] ([0,8,1] in
python) is defined by property offset(line 8 in Listing 5) and used as an
argument, which means the test would be launched eight times for each value in
the range. The final property set-var denotes that the variable stack-offset
would be assigned with the offset resulting a success test. For example, if
test return-to-wrong-call-site succeeds with argument 3, stack-offset is then
set to 3 and all other test cases using this variable would get an argument of
3. return-to-non-call-site is one of these cases (line 11). Its input argument
is therefore defined to use the value of stack-offset (line 13 in Listing 5).
Since return-to-wrong-call-site is listed as a prerequisite, the value of
stack-offset must have been assigned when return-to-non-call-site is tested.
In other words, the test scheduler and the JSON configuration ensure that
stack-offset is initiated before testing all the test cases depending on it.

###  6.2. Fight Against Compiler Optimization

Since test cases usually contain deliberately malformed code and the behavior
of the embedded assembly is commonly unexpected by compilers, we often find
that the attacking behavior of a test case is silently disarmed by compiler
optimizations. To avoid this, the test suite utilizes certain code patterns
and compiler directives to disable compiler optimization at precise locations
rather than forcefully applying compiler flags to disabling certain
optimization to all code. Here we describe some of the techniques utilized in
this test suite.

When a function is simple enough or seemly dead, with or without an embedded
assembly, compiler would trying to eliminate the function call by function
inlining, dead code elimination, constant propagation, or even return value
prediction. To prevent these optimizations, we use compiler attributes to
disable the inlining of key functions. Some key variables are declared
volatile to stop dead code elimination and return value prediction. In some
extreme cases, key values are fed to the test case through input arguments to
avoid constant propagation.

Compiler may choose to reorder instructions which breaks the assumptions in
certain test cases. One of such examples is the read-GOT test shown in Listing
6. Function get_got_func() on line 3 tries to obtain the address of the GOT
entry for rand(). A reliable way to get this address is to call rand()
immediately after get_got_func(). The address of the corresponding PLT
(procedure linkage table) entry is then saved as the return address for
get_got_func(). To our surprise, some Arm variants of GCC swap the calling of
rand() and the following dereference of the obtained GOT address (got), which
usually leads to a segment error, resulting in an incorrect detection of a
non-existent defense. To avoid such unexpected behavior, which might lead to
an incorrect detection of a non-existent defense, we add a compiler barrier on
line 5 to prevent instruction reorder on this particular location.

⬇

1// part of main() in read-GOT.cpp

2void *got = NULL;

3get_got_func(&got, offset);

4rand();

5COMPILER_BARRIER;

6return *(uintptr_t *)(got);

7

8// part of assembly.hpp

9#define COMPILER_BARRIER \

10 asm volatile("" : : : "memory")

Listing 6: Read GOT entries

Another interesting case is related to the hijacking of VTable pointers as
illustrated in Listing 7. The genuine VTable of the victim object orig is
replaced with the VTable from an irrelevant object fake on line 11. Supposing
this replacement is successful, the following orig->virtual_func() should call
the wrong function defined in Fake which exits with a code 0. However, the
compiler seemly believes orig->virtual_func() should always call the genuine
virtual function and replaces the VTable based indirect call with a direct
call by calculating an offset at compile time. The result is a failed test
case with an arguably successful attack. To avoid such compile time (possibly
link time as well) optimization, we hide the definition of the victim classes
to a shared library which is visible only at runtime.

⬇

1... // omit some code

2class Fake {

3public:

4 virtual void virtual_func() { exit(0); }

5 ... // omit some code

6};

7

8int main() {

9 Base *orig = new Base();

10 Fake *fake = new Fake();

11 write_vtable_pointer(orig, *((pvtable_t *)fake));

12 orig->virtual_func();

13 return 4;

14}

Listing 7: Fake VTable

###  6.3. Interpret Test Results

It is actually a big challenge to interpret the results of test cases. As
indicated by the examples shown in Section 6.2, unexpected compiler
optimizations may fail a test case and indicate the existence of an actually
non-existent defense. In fact, the reverse could happen as well, such as that
a defense may replace a dangling pointer with a random number pointing to a
still readable memory location. Although some coding techniques have been
applied to avoid these errors, removing all of them for all compilers is
difficult. When a test indeed fails due to a defense, it is also hard to tell
who implements this defense, the compiler, the runtime library, the kernel, or
the hardware. Considering a ROP attack as an example, it might be defeated by
software implemented canary or a shadow stack, or hardware supported pointer
authentication or memory tagging. In this test suite, two mechanisms are
utilized to reduce ambiguity and potentially warn about misinterpretation:

(1) Different exit codes are used for all exiting points of the program, such
as the code 4 used on line 13 of Listing 7. Code 0 is returned only when the
test is considered successful. Exit codes for known defenses are listed in the
configuration file, such as the code 16 in Listing 8. During the test, the
scheduler silents the output for all tests returning 0 or known codes while
highlighting the results for tests returning unexpected code as a way to
indicate the need for further investigation.

⬇

1{

2 ... // omit some code

3

4 "code-injection-stack": {

5 ... // omit some code

6 "results": { "16": "DEP protection" }

7 },

8

9 ... // omit some code

10}

Listing 8: Record expected exit codes

(2) When a test ends with an exception, the test suite tries to catch it with
extra checks. An example (case return-to-instruction-in-stack) is demonstrated
in Listing 9, which checks whether the return address can be hijacked to a
code injected on stack by a ROP attack. Similar to return-to-wrong-call-site
(Listing 2), the actual attack is launched by helper(m) on line 6. On most
platforms, such attack is easily thwarted by DEP. A segment error is thrown
immediately when the address of the injected code (m) is assigned to PC. This
behavior is caught by the outer pair of catch_exception() functions, which
specifically watches the SEGV_ACCERR error on address m. The test then exits
with code 16 to indicate DEP (with test termination by exception). In the rare
condition when stack is executable, the test would succeed with executing the
code on stack. To improve the platform independence and avoid recovery from a
corrupted call stack, the test injects a divide-by-zero instruction sequence
with the division instruction on m+4. The division would raise an exception of
type SIGFPE, which is caught by the inner pair of catch_exception(). The exit
code is then set to 0. All codes other than 0 and 16 would need further
investigation.

⬇

1// part of main() in

2// return-to-instruction-in-stack.cpp

3unsigned char m[] = DIVIDE_BY_0_CODE;

4begin_catch_exception(m, SEGV_ACCERR, 16, SIGSEGV);

5 begin_catch_exception(m+4, 0, 0, SIGFPE);

6 int rv = helper(m); // hijacted to m by ROP

7 end_catch_exception();

8end_catch_exception();

9exit(rv);

10

11// definition of begin_catch_exception

12// in assembly.hpp

13extern void begin_catch_exception(

14 void *addr, int si_code,

15 int exit_code, int si_signo);

Listing 9: Code injection (stack)

##  7\. Test Results

###  7.1. Testing Platforms

Table 2. Platform parameters Platform | Architecture | Processor | Operating System | Kernel | Compiler | C Library
---|---|---|---|---|---|---
Intel i7-3770 | x86-64 | i7-3770 | Ubuntu 16.04 | 4.15.0 | g++ 5.4.0 | GLIBC 2.23
Intel Xeon 8280 | x86-64 | Xeon 8280 | Ubuntu 18.04 | 5.4.0 | g++ 7.5.0 | GLIBC 2.27
Jetson Nano | ARMv8.0-A | Arm Cortex-A57 | Ubuntu 18.04 | 4.9.140 | g++ 7.5.0 | GLIBC 2.27
Raspberry Pi 4B | ARMv8.0-A | Arm Cortex-A72 | Ubuntu 20.10 | 5.8.0 | g++ 10.2.0 | GLIBC 2.32
HiFive Unleashed | RV64GC | SiFive u540 |  OpenEmbedded (SiFive Inc., 2021b) | 5.8.2 | g++ 10.2.0 | GLIBC 2.32
Spike | RV64GC | Spike 2020-Oct |  Buildroot (Xu, 2021) | 5.8.1 | g++ 10.1.0 | GLIBC 2.32

The test suite has been applied on six platforms using three different ISAs,
including Intel x86-64, Arm AArch64 and RISC-V. Intel x86-64 represents the
most used ISA in personal computers (PC) and servers. Arm AArch64 is currently
the most utilized ISA on smartphones. RISC-V (Editors Andrew Waterman and
Krste Asanović, RISC-V Foundation, 2019a) is a newly proposed and promising
open architecture which has been quickly adopted by the opensource community
and processor manufactures.

Table 2 illustrates the parameters of the six platforms. For Intel x86-64, a
relatively old i7-3770 with Ubuntu 16.04 and a recent Xeon 8280 with Ubuntu
18.04 are chosen as two representative platforms. We have managed to run the
test suite on two Arm single board computers which we have access to. One is a
Jetson Nano board with a Ubuntu 18.04 running on an Arm Cortex-A57 processor
and the other is a Raspberry Pi 4B board with a Ubuntu 20.10 running on an Arm
Cortex-A72 processor. Both processors comply with the ARMv8.0-A ISA but are
equipped with different operating systems. As for RISC-V, we have managed to
borrow a HiFive Unleashed board mounted with a SiFive u540 processor (SiFive
Inc., 2021a). Although HiFive Unleashed was released in 2018, it is still one
of the most powerful RISC-V computers ever produced. The operating system
running on the HiFive Unleashed is a standard pre-compilation of the
OpenEmbedded Linux (SiFive Inc., 2021b) recommended by SiFive inc. To
compensate for the lack of a relatively new RISC-V platform, we have compiled
and installed all the necessary libraries in a latest Linux kernel image using
buildroot (Xu, 2021) and run it on Spike (Waterman et al., 2021), which is
officially the golden reference model and simulator for the latest RISC-V ISA.

###  7.2. Comparison Between Platforms

Table 3. Test results with the default settings |  Intel i7-3770 |  Intel Xeon 8280 |  Jetson Nano |  Raspberry Pi 4B |  HiFive Unleashed |  Spike
---|---|---|---|---|---|---
Spatial safety (98) | 98 | 98 | 98 | 98 | 98 | 98
Temporal safety (13) | 13 | 13 | 13 | 9 | 9 | 9
Access control (3) | 3 | 2 | 2 | 2 | 2 | 3
Pointer integrity (5) | 5 | 4 | 4 | 4 | 5 | 5
CFI (41) | 28 | 28 | 28 | 28 | 28 | 24
Total (160) | 147 | 145 | 145 | 141 | 142 | 139

To answer the question _Q1_ raised in the introduction, our first set of tests
try to evaluate the security provided by different platforms. To produce a
fair comparison, we use the default setting on all platforms. The test suite
is compiled using the default GNU g++ compiler distributed by the OS with the
compiler flags set as -O2 -std=c++11 -Wall. Table 3 provides a summary of the
test results while detailed test results are revealed in Appendices A. Each
figure in the table denotes the number of test cases succeed in the specific
category. The success of a test case indicates that one type of
vulnerabilities is tested exploitable due to the lack of a memory check.
Consequently, a lower figure indicates better security as less tests succeed.
The name and the total number of cases of each category are provided in the
first column of Table 3.

####  7.2.1. Spatial safety

The results show that _there is currently no defense detected on all platforms
against the buffer overflow and memory spray attacks tested in this suite._
All test cases success without exception or error. This reflects that,
although numerous defense techniques have been proposed and some of them have
been incorporated into compilers in recent years, no defense is applied by
default probably due to performance concerns.

####  7.2.2. Temporal safety

Although most test cases related to temporal safety still pass on all
platforms, all heap-based read-after-free and write-before-reclaim tests fail
on platforms using GLIBC version newer than 2.32. A detailed look reveals that
the memory allocation algorithm comes along with the latest C library refills
the memory space with garbage during both the allocation and the release
processes, which effectively prevent data leakage after free and malicious
initialization for uninitiated variables. However, the new memory allocation
algorithm can still be forced to relocate the same object at the same memory
region released previously. UAF attacks using dangling pointers are still
viable. Meanwhile, stack based UAF attacks are rarely defended.

####  7.2.3. Access control

To our surprise, the tests indicate that ASLR is not deployed on Intel i7-3770
and Spike. With some further investigation, both platforms do support user-
level ASLR but the default compilers have the position independent executable
(PIE) feature disabled by default. The test case check-ASLR would fail as
excepted if it is compiled with -pie -fPIE. As for the test cases reading a
part of a function (read-func) or reading the content of a GOT (read-GOT), no
defense is detected on any of the platforms.

####  7.2.4. Pointer integrity

The test suite detects no defense against maliciously reading and modifying a
code pointer. Although compilers do warn about the arithmetic on a function
pointer, the test case still finishes without exception. There is no defense
found for protecting the VTable pointer as well. The only type of defense is
related to modifying an entry in the GOT. Although GOT can be read on all
platforms, they are placed on read-only pages on Intel Xeon 8280, Jetson Nano
and Raspberry Pi 4B complying with the relocation read-only (RELRO)
protection. Note that most Linux distributions support partial RELRO by
default. For the platforms where modify-GOT succeeds, the partial RELRO fails
to cover the entry for the library function (rand()) hijacked. The success of
modify-GOT on HiFive Unleashed and Spike indicates potential security issues
that have not been properly resolved in the latest RISC-V compilers.

####  7.2.5. Control-flow integrity

It seems like most CFI related defenses are disabled by default on all
platforms. All tests that hijack the return address to a code exiting in
memory succeed without exception. Only the code injection tests are prevented
by DEP as expected. One extra benefit of using the latest RISC-V ISA (on
Spike) is the failure of return-to-instruction-in-rodata which tries to
execute a code injected in constant data (read-only pages). Starting from
RISC-V privileged specification version 1.11 (Editors Andrew Waterman and
Krste Asanović, RISC-V Foundation, 2019b), read-only pages are no longer
executable by default as a separate “E” flag is defined in the page table
entry. Since the privileged specification adopted by HiFive Unleashed is
version 1.10, such feature is not supported yet and the test succeeds just
like on other platforms. However, to our surprise, the test still passes on
Spike when PIE is disabled (-no-pie). Some parts of the global constant data
might be put in the code section for a reduced memory footprint.

Similar to the tests for backward control-flow integrity, all tests that
hijack a function pointer succeed without a hitch except for the code
injection tests. The latest RISC-V provides the extra benefit of preventing
code execution on read-only but non-execution pages.

The current situation on the protection of VTables are worrying. It is found
that not only can attackers replace a Vtable with another existing one as
expected but also fabricate a table in writable memory sections such as stack,
heap and global data. There is no check on whether a VTable is stored on read-
only pages although compilers are doing so to avoid direct modification. The
only defense that has been found is a protection against malicious reuse of
VTable pointers enforced by the memory allocation algorithm. The memory
allocation algorithm does provide some protection against malicious VTable
replacement. According to the result of call-wrong-func-vtable-released, when
an object is released, the VTable pointer is zeroed by the memory allocator.
Reusing a VTable pointer from a released object is therefore prevented.

####  7.2.6. Summary of Results

Comparing across platforms, there is currently no significant difference for
the provided security with default settings while newer architectures (RISC-V)
using the latest compiler and runtime libraries show marginal security benefit
thanks to the support of R$\oplus$X in page tables and the protection against
common heap-based UAF attacks implemented in the newer C library.

###  7.3. Test with Different Flags and Compilers

It is common that a defense is added to a compiler but disabled by default due
to performance concerns. As described in _Q2_ and _Q3_ of the introduction,
application designer may wish to explore their choices on different
combination of compilers and defenses, and security researchers would like to
evaluate the strength of a certain defense on a specific platform. The test
suite would be a good tool for these kind of explorations.

For the investigation of different compiler flags, we have compiled and
installed the same version of the GCC compiler (version 10.3.0) and the C
library (GLIBC 2.32) on both Intel Xeon 8280 and Raspberry Pi 4B to cover
Intel x86-64 and Arm AArch64. No RISC-V platform is tested in this experiment
because some important defenses, such as the VTable verification (VTV) (Tice
et al., 2014), have not been ported to RISC-V yet. We have also installed a
latest LLVM 13 on Intel Xeon 8280 to explore the differences between
compilers.

Table 4. Available sets of compiler flags | compiler flags | compiler
---|---|---
Default | -O2 | GCC and LLVM
RELRO | -pie -fPIE -Wl,-z,relro,-z,now | GCC and LLVM
Stack protection | -Wstack-protector -fstack-protector-all | GCC and LLVM
VTV | -fvtable-verify=std | GCC only
CFI | -fvisibility=default -fsanitize=cfi | LLVM only
| -flto -fuse-ld=gold |
All | all the above | GCC and LLVM
ASan | -fsanitize=address –param=asan-stack=1 | GCC and LLVM
None | -z execstack -fno-stack-protector | GCC and LLVM
| (sysctl -w kernel.randomize_va_space=0) |

After a small survey on the available security features in both compilers,
Table 4 lists the sets of compiler flags being explored:

  * •

Default: Test using the latest compiler with the default compiler flags.

  * •

RELRO (RElocation Read-Only): Enforce the full protection of the GOT
(HockeyInJune, 2011).

  * •

Stack protection: Apply the stack smashing protection by inserting a canary on
the stack (Cowan et al., 2003b).

  * •

VTV: Apply the GCC variant of defense against COOP attacks.

  * •

CFI: Similar to VTV, apply the LLVM variant of defense against all forward CFI
attacks (Tice et al., 2014).

  * •

All: Apply all the above defenses.

  * •

ASan: Enable the full address sanitizer at runtime.

  * •

None: Deliberately disable all proctections, including the one applied by the
kernel, such as ASLR.

Some defenses are supported in compilers by not evaluated. Intel CET (Intel
Corporation, 2019) is supported by the latest compiler but the hardware is not
ready yet. Intel MPX (Ramakesavan et al., 2016) has been removed from GCC in
recent years. Arm PA (ARM, 2021) and Arm MTE (Bannister, 2019) are supported
in LLVM but unfortunately we do not have access to a compatible Arm platform
for the test. Stack clash protection has been tested but does not affect the
result.

Table 5. Test results with different flags and compilers |  Intel Xeon 8280 (GCC) |  Raspberry Pi 4B (GCC) |  Intel Xeon 8280 (LLVM)
---|---|---|---
Default | 142 | 142 | 142
RELRO | 141 | 141 | 140
Stack protection | 141 | 140 | 141
VTV | 136 | 136 | N/A
CFI | N/A | N/A | 141
All | 134 | 133 | 138
ASan | 8 | 8 | 21
None | 155 | 147 | 154

Table 5 reveals the total number of passed test cases with different flags and
compilers while more detailed results are provide in Appendix B.

For the test of using different compiler flags, it is found that enabling
security features does defeat the targeted types of attacks and there is no
significant difference on the effectiveness of these features on different
platforms, at least for the platforms we have tested. For the default setting
using GCC, Intel Xeon 8280 and Raspberry Pi 4B have exactly the same result of
142 passed tests. Compared with the results in Table 3, the hijacking of GOT
entry passes on both platforms as the full protection of GOT is deliberately
disabled in the compilation of the compiler, and four UAF on heap tests fail
on Intel Xeon 8280 thanks to the newer C library. By enabling the full GOT
protection (case RELRO), the total number of tests passed reduces by one on
both platforms. Enabling stack protection has very limited protection on both
platforms because most ROP attacks can pinpoint the location of the return
address and modify it without touching the canary. Only the test trying to
fake a call frame fails on both platform as it cannot create a correct canary.
As the only extra benefit, The test for return-to-libc (Pincus and Baker,
2004) fails on Raspberry Pi 4B as the checking of canary in the function
epilogue disturbs the malicious manipulation of function arguments. VTV has
exactly the same effect on both platforms that six COOP attacks have been
stopped. Any attack that hijacks a VTable to an injected one or even one
failing the class hierarchy analysis (Jang et al., 2014) is detected and
stopped but it is still possible to malicious reuse the VTables from
relatives, such as the parent, child and sibling classes. When all the
mentioned security features are enabled, the protection is simply combined
together with 26 tests fail on Intel Xeon 8280 and 27 tests failed on
Raspberry Pi 4B. Address sanitizer (ASan) is a well known debugging tool that
is normally disabled at production-phase. We have also done an experiment to
see its efficiency. The result is impressive as only eight tests remain
successful while all other 152 tests fail on both platforms. The remaining
tests include two access control tests (read-func and read-GOT) and six UAF
attacks on stack. We think the latter could be included by future address
sanitizers. Finally, when all protection features are deliberately disabled
(the none case), including ASLR and DEP (execution on stack), only five and
thirteen tests fail on Intel Xeon 8280 and Raspberry Pi 4B respectively. All
the failed tests on Intel Xeon 8280 are heap-based UAF attacks thwarted by the
latest C library. Eight extra tests fails on Raspberry Pi 4B because code
injection on heap and global data is still prohibited on Arm AArch64 even when
execution on stack is allowed. As a result, Arm AArch64 is marginally safer
than Intel x86-64. Nearly the same result is observed on the two RISC-V
platforms excepted for VTV as it is not yet ported by GCC.

The test suite is portable enough that no change is needed when switching to
LLVM using the default setting. ASLR is found disabled as the distribution
provided LLVM produces non-PIE executables by default but arithmetic
operations on function pointers are explicitly disallowed (Chen and Chisnall,
2019). The result is the same number of tests passed for the default case
compared with GCC. Enabling RELRO not only activates the full GOT protection
but also ASLR as PIE is enabled. The stack protection provided by LLVM has
exactly the same effect as GCC. Only the test trying to fake a call frame
fails. To our surprise, enabling CFI has almost no protection according to the
result; however, we believe the test suite in the current form cannot provide
an accurate estimation for this protection. The implementation of CFI
protection on LLVM resorts to the link time optimization and requires
visibility to all definitions of virtual classes at link time. This
requirement forces the static linking of class definition mentioned in Section
6.2 rather than dynamic linking through a shared library as in GCC. As a
result, we suspect that all the illegal modifications on VTable pointers and
function pointers are taken as valid operations by the link time analysis.
Consequently, enabling all the mentioned protection using LLVM results higher
number of passed tests than GCC. LLVM’s address sanitizer (ASan) is seemly
less effective than it on GCC. It catches all ROP and COOP attacks but let go
COP and JOP attacks. It also fails to detect the modification on GOT entries.
On the good side, it catches nearly all UAF attacks, including the UAF on
stack attacks missed by GCC’s ASan. Finally, when all protections are disabled
(case none), LLVM and GCC have the same result except that the test applying
arithmetic operations on function pointers fails to compile on LLVM.

##  8\. Discussion

###  8.1. Related Work

Computer architectures have for decades been constantly improved to satisfy
our thirst for performance. To quantitatively evaluate the performance gain
brought by each improvement, a number of test suites have been carefully
designed and widely utilized. LINPACK (Dongarra et al., 1979) was introduced
in the 1970s to measure the computing power for numerical linear algebra and
is still used to rank today’s supercomputers. Dhrystone (Weicker, 1984)
provides a suitable indicator for the general integer performance of
computers. CoreMark (Gal-On and Levy, 2012) specializes in the performance of
micro-controllers while the more powerful general-purpose computers are
normally compared using the SPEC benchmark suite (Henning, 2006). PARSEC
(Bienia et al., 2008) concentrates on the performance of shared memory and
multi-thread applications.

Using a test suite to measure the security of a certain system is not new
either. They began to appear in 2005, when two papers (Kratkiewicz and
Lippmann, 2005; Zhivich et al., 2005) proposed to use corpora of small and
synthetic buffer overflow attacks (291 in (Kratkiewicz and Lippmann, 2005) and
55 in (Zhivich et al., 2005)) to test whether they can be detected by the then
state-of-the-art software defenses, such as CRED (Ruwase and Lam, 2004) and
CCured (Necula et al., 2005). Later in 2006, BASS (Poe and Li, 2006) adopted
an approach similar to the SPEC CPU test suite. It contained seven relatively
large test cases embedded with different types of vulnerabilities related to
memory spatial safety. It then provided a framework for the automatic
generation of attacks exploiting the embedded vulnerabilities. To our best
knowledge, BASS was the first attempt for measuring the security of a computer
platform while its scope is limited to several types of memory spatial errors.
RIPE (Wilander et al., 2011) is probably the most utilized security test suite
for defenses related to memory safety. By exhausting the possible combinations
of several attacking techniques, RIPE is able to cover 850 forms of buffer
overflow and ROP attacks. It has already been extensively utilized to verify
the strength of various hardware supported defenses against control-flow
attacks (Kuznetsov et al., 2014; Song et al., 2016). However, the fact that it
uses as many as 850 test cases to cover only buffer overflow and ROP attacks
indicates that reaching a full coverage by exhausting all possible
combinations of vulnerabilities and attack techniques is unrealistic. This
partially motivates us to produce a new testing framework.

The design of security test suites has been revived in recent years. The
concept of a CPU security benchmark was introduced in (Zhu et al., 2018) which
proposed several testing techniques also utilized in our test suite.
Unfortunately, the benchmark is not available for comparison, no detail was
revealed with regard to its internal structure, and no result is provided in
the paper. CONFIRM (Xu et al., 2019) was a recently proposed test suite to
evaluate the compatibility and applicability of various control-flow integrity
defenses for different applications but it provides very little information on
the security provided by the evaluated defenses. Motivated by the lack of
security evaluation by CONFIRM, CBench (Li et al., 2020) was introduced to
measure the gap between the practical security strength and the claimed
security of different control-flow integrity defenses. It followed a similar
approach with BASS. A total of eighteen vulnerable programs were provided in
seven categories. By running the vulnerable programs and attacking them at
runtime, CBench measures the security level of the applied CFI defense.
Compared with the test suite proposed in this paper, CBench experiments with
full-fledged attacks while overlooking the relationship between individual
vulnerabilities. It concentrates on evaluating the strength of the defense
mechanisms themselves but not the platform implementing them. Consequently,
CBench does not need to support multiple platforms and runs only on x86-64.

###  8.2. Future Work

The testing framework proposed in this paper is far from finished. It would
need some significant extensions to become a full-fledged test suite measuring
the security supported by the processor hardware.

####  8.2.1. Missing test cases for memory safety

One category of vulnerabilities missing in the current test suite is related
to data racing in multi-threaded applications (Li et al., 2020). When the
integrity check on a pointer and the following pointer dereferencing do not
operate atomically, an attacker running on a parallel thread might hijack the
pointer just after the integrity check but before the pointer dereferencing.
Currently, all test cases are single threaded. Supporting multi-threaded tests
for data racing conditions is one of our immediate future works.

Test cases for DOP attacks (Hu et al., 2016) are being gradually added. The
number of test cases for type confusion and malicious initialization of
uninitiated variables will be substantially increased to cover all
possibilities. It is also reported that a significant portion of DOP attacks
violate the visibility scope of variables (Nyman et al., 2019). Applying fine-
grained compartmentalization of data memory to enforce the visibility scope of
variables is a promising defense against DOP attacks and is not yet fully
covered by the test suite.

####  8.2.2. Missing test cases for other types of safety

Although important, memory safety is only one of the many categories of
security that can be supported by the processor hardware. At least two other
categories of defenses should be included in this test suite: hardware
supported defenses against the _software-based side-channel attacks_ and the
_transient attacks_ (Canella et al., 2019a).

The software-based side-channel attacks include all side-channel or covert
channel attacks that can be launched purely by software. They include both the
conflict-based (Yan et al., 2017) and flush-based (Yarom and Falkner, 2014)
cache side-channel attacks. They contain also the attacks causing timing
channels inside the processor core by maliciously competing resources, such as
the recent port smash attack (Aldaya et al., 2019). Although cache side-
channel attacks against the traditional level-one caches are thoroughly
evaluated by (Deng et al., 2020), this test suite has not covered the more
prevalent attacks against the last-level caches, cannot be easily ported to
other non-x86 architectures, and still misses coverage on the new variants of
side-channel attacks (Bourgeat et al., 2020; Purnal et al., 2021; Song et al.,
2021) targeting the recently revived randomized caches (Qureshi, 2019; Werner
et al., 2019).

The term transient execution attack was first introduced in (Canella et al.,
2019a) and used to summarize all attacks that exploit the speculative
execution in modern out-of-order processors. It includes all variants of the
Meltdown (Lipp et al., 2018) and the Spectra (Kocher et al., 2019) attacks.
New variants of transient execution attacks (Van Bulck et al., 2018; Canella
et al., 2019b; Van Bulck et al., 2020) are still being found in recent years
along with various new defense proposals (Saileshwar and Qureshi, 2019;
Ainsworth and Jones, 2020; Loughlin et al., 2021). A proper test suite to
systematically evaluate the remaining attack vectors left by the different
defenses is urgently needed.

####  8.2.3. Vulnerability scoring system

We still need a vulnerability scoring system to interpret the test result and
translate it into a score quantitively indicating the security supported by
the processor.

Every vulnerability recorded in the common vulnerabilities and exposures (CVE)
database (MITRE, 2021b) is assigned with a vulnerability score measured by the
common vulnerability scoring system (CVSS) (FIRST.Org, Inc., 2021). However,
we cannot directly adopt this scoring system for evaluating architectural
vulnerabilities as the metric used is strictly software oriented. We have also
investigated the common weakness enumeration (CWE) (MITRE, 2021a) and found 58
related CWEs. By averaging the CVSS scores of the CVEs related to each CWE, we
might be able to calculate an equivalent score for each test case using the
existing scores from CVEs. Unfortunately, most of the 58 CWEs are linked to
the test cases regarding to spatial and temporal safety of memory, while most
of the control flow related test cases have no connected CWE. As a result, we
cannot directly borrow the existing CVSS to score the architectural
vulnerabilities. This is another open challenge that requires solutions.

##  9\. Conclusion

A comprehensive and cross-platform test suite has been produced to measure
memory safety. It is also our first step in producing a fully-fledged test
suite to measure the security supported by the processor hardware. The initial
test suite currently contains 160 test cases covering spatial and temporal
safety of memory, memory access control, pointer integrity and control-flow
integrity. Each type of vulnerabilities and their related defenses have been
individually evaluated by one or more test cases. To show its usefulness, the
test suite has been ported to six platforms using three different ISAs,
including Intel x86-64, Arm AArch64 and RISC-V. According to our experiment
results, most memory safety vulnerabilities are still exploitable with the
default compiler settings. Enabling extra security features in compilers does
defeat the targeted types of attacks. Although address sanitizer is often a
tool for debugging and not a practical production-phase protection measure, it
is indeed effective in catching attacks. Comparing across platforms, newer
architectures using the latest compiler and runtime libraries show marginal
security benefit, at least for the platforms we have tested. Comparing between
compilers, LLVM has similar security support with GCC but its address
sanitizer is seemly weaker.

## Availability

The benchmark has been open sourced on GitHub at
https://github.com/comparch-security/cpu-sec-bench.

## Acknowledgment

Yuhui Zhang has contributed the early versions of this benchmark. The HiFive
Unleashed board was kindly borrowed from Xiongfei Guo. This work was partially
supported by the National Natural Science Foundation of China under grant No.
61802402, the CAS Pioneer Hundred Talents Program, and internal grants from
the Institute of Information Engineering, CAS. Any opinions, findings,
conclusions, and recommendations expressed in this paper are those of the
authors and do not necessarily reflect the views of the funding parties.

## References

  * (1)
  * Abadi et al. (2009) Martín Abadi, Mihai Budiu, Úlfar Erlingsson, and Jay Ligatti. 2009.  Control-flow integrity principles, implementations, and applications.  _ACM Transactions on Information and System Security_ 13, 1, Article 4 (Nov. 2009).
  * Aga and Austin (2019) Misiker Tadesse Aga and Todd M. Austin. 2019.  Smokestack: Thwarting DOP attacks with runtime stack layout randomization. In _IEEE/ACM International Symposium on Code Generation and Optimization_. IEEE, 26–36.
  * Ainsworth and Jones (2020) Sam Ainsworth and Timothy M. Jones. 2020.  MuonTrap: Preventing cross-domain spectre-like attacks by capturing speculative state. In _ACM/IEEE Annual International Symposium on Computer Architecture_. IEEE, 132–144.
  * Akritidis (2010) Periklis Akritidis. 2010\.  Cling: A memory allocator to mitigate dangling pointers. In _USENIX Security Symposium_. USENIX Association, 177–192.
  * Aldaya et al. (2019) Alejandro Cabrera Aldaya, Billy Bob Brumley, Sohaib ul Hassan, Cesar Pereida García, and Nicola Tuveri. 2019.  Port contention for fun and profit. In _IEEE Symposium on Security and Privacy_. IEEE, 870–887.
  * Aleph One (1996) Aleph One. 1996\.  Smashing the stack for fun and profit.  _Phrack_ 49 (Nov. 1996).
  * ARM (2021) ARM. 2021.  Pointer authentication support in LLVM toolchain.  https://developer.arm.com/tools-and-software/open-source-software/developer-tools/llvm-toolchain/pointer-authentication.
  * Arora et al. (2006) Divya Arora, Anand Raghunathan, Srivaths Ravi, and Niraj K. Jha. 2006.  Architectural support for safe software execution on embedded processors. In _International Conference on Hardware/Software Codesign and System Synthesis_. ACM, 106–111.
  * Bannister (2019) Steve Bannister. 2019\.  Memory tagging extension: Enhancing memory safety through architecture.  https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety.
  * Bhatkar and DuVarney (2005) Sandeep Bhatkar and Daniel C. DuVarney. 2005.  Efficient techniques for comprehensive protection from memory error exploits. In _USENIX Security Symposium_. USENIX Association.
  * Bienia et al. (2008) Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008.  The PARSEC benchmark suite: Characterization and architectural implications. In _International Conference on Parallel Architectures and Compilation Techniques_ , Andreas Moshovos, David Tarditi, and Kunle Olukotun (Eds.). ACM, 72–81.
  * Bletsch et al. (2011) Tyler K. Bletsch, Xuxian Jiang, Vincent W. Freeh, and Zhenkai Liang. 2011.  Jump-oriented programming: A new class of code-reuse attack. In _ACM Symposium on Information, Computer and Communications Security_. ACM, 30–40.
  * Boland and Black (2012) Tim Boland and Paul E. Black. 2012.  Juliet 1.1 C/C++ and Java test suite.  _Computer_ 45, 10 (Oct. 2012), 88–90.
  * Bourgeat et al. (2020) Thomas Bourgeat, Jules Drean, Yuheng Yang, Lillian Tsai, Joel Emer, and Mengjia Yan. 2020\.  CaSA: End-to-end quantitative security analysis of randomly mapped caches. In _IEEE/ACM International Symposium on Microarchitecture_. 1110–1123.
  * Brash (2016) David Brash. 2016\.  Armv8-A architecture: 2016 additions.  https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/armv8-a-architecture-2016-additions.
  * Burow et al. (2017) Nathan Burow, Scott A. Carr, Joseph Nash, Per Larsen, Michael Franz, Stefan Brunthaler, and Mathias Payer. 2017.  Control-flow integrity: Precision, security, and performance.  _Comput. Surveys_ 50, 1 (2017), 16:1–16:33.
  * Burow et al. (2019) Nathan Burow, Xinping Zhang, and Mathias Payer. 2019\.  SoK: Shining light on shadow stacks. In _IEEE Symposium on Security and Privacy_. IEEE, 985–999.
  * c0ntex (2006) c0ntex. 2006.  How to hijack the global offset table with pointers for root shells.  https://www.exploit-db.com/papers/13237.
  * Canella et al. (2019a) Claudio Canella, Jo Van Bulck, Michael Schwarz, Moritz Lipp, Benjamin von Berg, Philipp Ortner, Frank Piessens, Dmitry Evtyushkin, and Daniel Gruss. 2019a.  A systematic evaluation of transient execution attacks and defenses. In _USENIX Security Symposium_. USENIX Association, 249–266.
  * Canella et al. (2019b) Claudio Canella, Daniel Genkin, Lukas Giner, Daniel Gruss, Moritz Lipp, Marina Minkin, Daniel Moghimi, Frank Piessens, Michael Schwarz, Berk Sunar, Jo Van Bulck, and Yuval Yarom. 2019b.  Fallout: Leaking data on meltdown-resistant CPUs. In _ACM SIGSAC Conference on Computer and Communications Security_. ACM, 769–784.
  * Carlini and Wagner (2014) Nicholas Carlini and David A. Wagner. 2014.  ROP is still dangerous: Breaking modern defenses. In _USENIX Security Symposium_. USENIX Association, 385–399.
  * Chang et al. (2008) Walter Chang, Brandon Streiff, and Calvin Lin. 2008\.  Efficient and extensible security enforcement using dynamic data flow analysis. In _ACM Conference on Computer and Communications Security_. ACM, 39–50.
  * Chen et al. (2005) Shuo Chen, Jun Xu, and Emre Can Sezer. 2005.  Non-control-data attacks are realistic threats. In _USENIX Security Symposium_. USENIX Association.
  * Chen and Chisnall (2019) Tony Chen and David Chisnall. 2019.  _Pointer tagging for memory safety_.  Technical Report MSR-TR-2019-17. Microsoft.  https://www.microsoft.com/en-us/research/publication/pointer-tagging-for-memory-safety/.
  * Conover and w00w00 Security Team (1999) Matt Conover and w00w00 Security Team. 1999.  w00w00 on heap vverflows.  http://hamsa.cs.northwestern.edu/media/readings/heap_overflows.pdf.
  * Conti et al. (2015) Mauro Conti, Stephen Crane, Lucas Davi, Michael Franz, Per Larsen, Marco Negro, Christopher Liebchen, Mohaned Qunaibit, and Ahmad-Reza Sadeghi. 2015\.  Losing control: On the effectiveness of control-flow integrity under stack attacks. In _ACM SIGSAC Conference on Computer and Communications Security_. ACM, 952–963.
  * Cowan et al. (2003a) Crispin Cowan, Steve Beattie, John Johansen, and Perry Wagle. 2003a.  PointGuard™: Protecting pointers from buffer overflow vulnerabilities. In _USENIX Security Symposium_. USENIX Association.
  * Cowan et al. (2003b) Crispin Cowan, Perry Wagle, Calton Pu, Steve Beattie, and Jonathan Walpole. 2003b.  Buffer overflows: Attacks and defenses for the vulnerability of the decade. In _Foundations of Intrusion Tolerant Systems_. 227–237.
  * Dang et al. (2015) Thurston H. Y. Dang, Petros Maniatis, and David A. Wagner. 2015\.  The performance cost of shadow stacks and stack canaries. In _ACM Symposium on Information, Computer and Communications Security_. ACM, 555–566.
  * Das et al. (2019) Sourav Das, R. Harikrishnan Unnithan, Arjun Menon, Chester Rebeiro, and Kamakoti Veezhinathan. 2019.  SHAKTI-MS: a RISC-V processor for memory safety in C. In _ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems_. ACM, 19–32.
  * Davi et al. (2014) Lucas Davi, Ahmad-Reza Sadeghi, Daniel Lehmann, and Fabian Monrose. 2014.  Stitching the gadgets: On the ineffectiveness of coarse-grained control-flow integrity protection. In _USENIX Security Symposium_. USENIX Association, 401–416.
  * Deng et al. (2020) Shuwen Deng, Wenjie Xiong, and Jakub Szefer. 2020\.  A benchmark suite for evaluating caches’ vulnerability to timing attacks. In _International Conference on Architectural Support for Programming Languages and Operating Systems_. ACM, 683–697.
  * Devietti et al. (2008) Joe Devietti, Colin Blundell, Milo M. K. Martin, and Steve Zdancewic. 2008.  Hardbound: Architectural support for spatial safety of the C programming language. In _International Conference on Architectural Support for Programming Languages and Operating Systems_. ACM, 103–114.
  * Dhawan et al. (2014) Udit Dhawan, Nikos Vasilakis, Raphael Rubin, Silviu Chiricescu, Jonathan M. Smith, Thomas F. Knight, Benjamin C. Pierce, and André DeHon. 2014.  PUMP: A programmable unit for metadata processing. In _Workshop on Hardware and Architectural Support for Security and Privacy_. ACM Press.
  * Ding et al. (2017) Ren Ding, Chenxiong Qian, Chengyu Song, William Harris, Taesoo Kim, and Wenke Lee. 2017\.  Efficient protection of path-sensitive control security. In _USENIX Security Symposium_. USENIX Association, 131–148.
  * Dongarra et al. (1979) Jack J. Dongarra, Cleve Barry Moler, James R. Bunch, and Gilbert W. Stewart. 1979.  _LINPACK users’ guide_.  SIAM.
  * Editors Andrew Waterman and Krste Asanović, RISC-V Foundation (2019a) Editors Andrew Waterman and Krste Asanović, RISC-V Foundation 2019a.  _The RISC-V Instruction Set Manual. Volume I: Unprivileged ISA Version 20191213_.  Editors Andrew Waterman and Krste Asanović, RISC-V Foundation.  https://riscv.org/technical/specifications/.
  * Editors Andrew Waterman and Krste Asanović, RISC-V Foundation (2019b) Editors Andrew Waterman and Krste Asanović, RISC-V Foundation 2019b.  _The RISC-V Instruction Set Manual. Volume II: Privileged Architecture Version 20190608_.  Editors Andrew Waterman and Krste Asanović, RISC-V Foundation.  https://riscv.org/technical/specifications/.
  * Evans et al. (2015) Isaac Evans, Fan Long, Ulziibayar Otgonbaatar, Howard E. Shrobe, Martin C. Rinard, Hamed Okhravi, and Stelios Sidiroglou-Douskos. 2015\.  Control jujutsu: On the weaknesses of fine-grained control flow integrity. In _ACM SIGSAC Conference on Computer and Communications Security_. ACM, 901–913.
  * FIRST.Org, Inc. (2021) FIRST.Org, Inc. 2021\.  Common vulnerability scoring system version 3.1.  https://www.first.org/cvss/.
  * Fogh and Ertl (2018) Anders Fogh and Christopher Ertl. 2018.  Wrangling the Ghost: An Inside Story of Mitigating Speculative Execution Side Channel Vulnerabilities. In _Black Hat USA_.
  * Forrest et al. (1997) Stephanie Forrest, Anil Somayaji, and David H. Ackley. 1997\.  Building diverse computer systems. In _Workshop on Hot Topics in Operating Systems_. IEEE Computer Society, 67–72.
  * Gal-On and Levy (2012) Shay Gal-On and Markus Levy. 2012.  Exploring CoreMarkTM — A benchmark maximizing simplicity and efficacy.  https://www.eembc.org/techlit/articles/coremark-whitepaper.pdf.
  * Göktas et al. (2014) Enes Göktas, Elias Athanasopoulos, Herbert Bos, and Georgios Portokalidis. 2014.  Out of control: Overcoming control-flow integrity. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 575–589.
  * Henning (2006) John L. Henning. 2006\.  SPEC CPU2006 benchmark descriptions.  _SIGARCH Computer Architecture News_ 34, 4 (2006), 1–17.
  * HockeyInJune (2011) HockeyInJune. 2011\.  RELRO: RELocation Read-Only.  http://blog.isis.poly.edu/exploitation%20mitigation%20techniques/exploitation%20techniques/2011/06/02/relro-relocation-read-only/.
  * Hu et al. (2016) Hong Hu, Shweta Shinde, Sendroiu Adrian, Zheng Leong Chua, Prateek Saxena, and Zhenkai Liang. 2016\.  Data-oriented programming: On the expressiveness of non-control data attacks. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 969–986.
  * Intel Corporation (2019) Intel Corporation. 2019\.  _Control-flow enforcement technology specification_.  Intel Corporation.
  * Jang et al. (2014) Dongseok Jang, Zachary Tatlock, and Sorin Lerner. 2014\.  SafeDispatch: Securing C++ virtual calls from Memory corruption attacks. In _Annual Network and Distributed System Security Symposium_. The Internet Society.
  * Kocher et al. (2019) Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, and Yuval Yarom. 2019\.  Spectre attacks: Exploiting speculative execution. In _IEEE Symposium on Security and Privacy_. IEEE, 1–19.
  * Kratkiewicz and Lippmann (2005) Kendra Kratkiewicz and Richard Lippmann. 2005.  Using a diagnostic corpus of C programs to evaluate buffer overflow detection by static analysis tools. In _Workshop on the Evaluation of Software Defect Detection Tools_ (Chicago, IL).
  * Kuznetsov et al. (2014) Volodymyr Kuznetsov, László Szekeres, Mathias Payer, George Candea, R. Sekar, and Dawn Song. 2014.  Code-pointer integrity. In _USENIX Conference on Operating Systems Design and Implementation_ (Broomfield, CO). USENIX Association, 147–163.
  * Kwon et al. (2013) Albert Kwon, Udit Dhawan, Jonathan M. Smith, Thomas F. Knight Jr., and André DeHon. 2013.  Low-fat pointers: Compact encoding and efficient gate-level implementation of fat pointers for spatial safety and capability-based security. In _ACM SIGSAC Conference on Computer and Communications Security_. ACM, 721–732.
  * Lee et al. (2015) Byoungyoung Lee, Chengyu Song, Yeongjin Jang, Tielei Wang, Taesoo Kim, Long Lu, and Wenke Lee. 2015.  Preventing use-after-free with dangling pointers nullification. In _Annual Network and Distributed System Security Symposium_. The Internet Society.
  * Li et al. (2020) Yuan Li, Mingzhe Wang, Chao Zhang, Xingman Chen, Songtao Yang, and Ying Liu. 2020\.  Finding cracks in shields: On the security of control flow integrity mechanisms. In _ACM SIGSAC Conference on Computer and Communications Security_. ACM, 1821–1835.
  * Lipp et al. (2018) Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Anders Fogh, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, Yuval Yarom, and Mike Hamburg. 2018\.  Meltdown: Reading kernel memory from user space. In _USENIX Security Symposium_. USENIX Association, 973–990.
  * Liu et al. (2016) Fangfei Liu, Qian Ge, Yuval Yarom, Frank McKeen, Carlos V. Rozas, Gernot Heiser, and Ruby B. Lee. 2016.  CATalyst: Defeating last-level cache side channel attacks in cloud computing. In _International Symposium on High Performance Computer Architecture_. 406–418.
  * Loughlin et al. (2021) Kevin Loughlin, Ian Neal, Jiacheng Ma, Elisa Tsai, Ofir Weisse, Satish Narayanasamy, and Baris Kasikci. 2021.  DOLMA: Securing speculation with the principle of transient non-observability. In _USENIX Security Symposium_. USENIX Association.
  * Mashtizadeh et al. (2015) Ali José Mashtizadeh, Andrea Bittau, Dan Boneh, and David Mazières. 2015.  CCFI: Cryptographically enforced control flow integrity. In _ACM SIGSAC Conference on Computer and Communications Security_. ACM, 941–951.
  * MITRE (2021a) MITRE. 2021a.  Common weakness enumeration — A community-developed list of software & hardware weakness types.  https://cwe.mitre.org/index.html.
  * MITRE (2021b) MITRE. 2021b.  CVE — Common vulnerabilities and exposures.  https://cve.mitre.org/.
  * Nagarakatte et al. (2012) Santosh Nagarakatte, Milo M. K. Martin, and Steve Zdancewic. 2012\.  Watchdog: Hardware for safe and secure manual memory management and full memory safety. In _Annual International Symposium on Computer Architecture_. 189–200.
  * Nagarakatte et al. (2014) Santosh Nagarakatte, Milo M. K. Martin, and Steve Zdancewic. 2014\.  WatchdogLite: Hardware-accelerated compiler-based pointer checking. In _Annual IEEE/ACM International Symposium on Code Generation and Optimization_. ACM, 10.
  * Nagarakatte et al. (2009) Santosh Nagarakatte, Jianzhou Zhao, Milo M.K. Martin, and Steve Zdancewic. 2009.  SoftBound: Highly compatible and complete spatial memory safety for C. In _ACM SIGPLAN Conference on Programming Language Design and Implementation_. ACM, 245–258.
  * Necula et al. (2005) George C. Necula, Jeremy Condit, Matthew Harren, Scott McPeak, and Westley Weimer. 2005.  CCured: Type-safe retrofitting of legacy software.  _ACM Transactions on Programming Languages and Systems_ 27, 3 (May 2005), 477–526.
  * Nellis (2018) Stephen Nellis. 2018\.  Intel did not tell U.S. cyber officials about chip flaws until made public.  https://www.reuters.com/article/us-cyber-intel/intel-did-not-tell-u-s-cyber-officials-about-chip-flaws-until-made-public-idUSKCN1G62PS.
  * Novark and Berger (2010) Gene Novark and Emery D. Berger. 2010.  DieHarder: Securing the heap. In _ACM Conference on Computer and Communications Security_. ACM, 573–584.
  * Nyman et al. (2019) Thomas Nyman, Ghada Dessouky, Shaza Zeitouni, Aaro Lehikoinen, Andrew Paverd, N. Asokan, and Ahmad-Reza Sadeghi. 2019\.  HardScope: Hardening embedded systems against data-oriented attacks. In _Annual Design Automation Conference_. ACM, 63.
  * Open Source Security, Inc (2021) Open Source Security, Inc. 2021\.  Grsecurity®.  https://grsecurity.net/.
  * Pincus and Baker (2004) Jonathan D. Pincus and Brandon Baker. 2004.  Beyond stack smashing: Recent advances in exploiting buffer overruns.  _IEEE Security & Privacy_ 2, 4 (2004), 20–27.
  * Poe and Li (2006) James Poe and Tao Li. 2006\.  BASS: A benchmark suite for evaluating architectural security systems.  _SIGARCH Computer Architecture News_ 34, 4 (Sept. 2006), 26–33.
  * Pomonis et al. (2019) Marios Pomonis, Theofilos Petsios, Angelos D. Keromytis, Michalis Polychronakis, and Vasileios P. Kemerlis. 2019\.  Kernel protection against just-in-time code reuse.  _ACM Transaction on Privacy and Security_ 22, 1 (2019), 5:1–5:28.
  * Purnal et al. (2021) Antoon Purnal, Lukas Giner, Daniel Gruss, and Ingrid Verbauwhede. 2021.  Systematic analysis of randomization-based protected cache architectures. In _IEEE Symposium on Security and Privacy_.
  * Qin et al. (2005) Feng Qin, Shan Lu, and Yuanyuan Zhou. 2005.  SafeMem: Exploiting ECC-memory for detecting memory leaks and memory corruption during production runs. In _International Conference on High-Performance Computer Architecture_. IEEE Computer Society, 291–302.
  * Qureshi (2019) Moinuddin K. Qureshi. 2019\.  New attacks and defense for encrypted-address cache. In _International Symposium on Computer Architecture_. ACM, 360–371.
  * Ramakesavan et al. (2016) Ramu Ramakesavan, Dan Zimmerman, Pavithra Singaravelu, George Kuan, Brian Vajda, Scott Gibbons, and Gautham Beeraka. 2016.  _Intel® memory protection extensions enabling guide_.
  * Roessler and DeHon (2018) Nick Roessler and André DeHon. 2018.  Protecting the stack with metadata policies and tagged Hardware. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 478–495.
  * Ruwase and Lam (2004) Olatunji Ruwase and Monica S. Lam. 2004.  A practical dynamic buffer overflow detector. In _Network and Distributed System Security Symposium_. The Internet Society.
  * Saileshwar and Qureshi (2019) Gururaj Saileshwar and Moinuddin K. Qureshi. 2019.  CleanupSpec: An “undo” approach to safe speculation. In _Annual IEEE/ACM International Symposium on Microarchitecture_. ACM, 73–86.
  * Sarbinowski et al. (2016) Pawel Sarbinowski, Vasileios P. Kemerlis, Cristiano Giuffrida, and Elias Athanasopoulos. 2016\.  VTPin: Practical VTable hijacking protection for binaries. In _Annual Conference on Computer Security Applications_ (Los Angeles, California, USA). ACM, 448–459.
  * Schuster et al. (2015) Felix Schuster, Thomas Tendyck, Christopher Liebchen, Lucas Davi, Ahmad-Reza Sadeghi, and Thorsten Holz. 2015.  Counterfeit object-oriented programming: On the difficulty of preventing code reuse attacks in C++ applications. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 745–762.
  * Serebryany et al. (2012) Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012.  AddressSanitizer: A fast address sanity checker. In _USENIX Annual Technical Conference_. USENIX Association, 309–318.
  * Serebryany and Herle (2019) Kostya Serebryany and Sudhi Herle. 2019.  Adopting the Arm memory tagging extension in Android.  https://security.googleblog.com/2019/08/adopting-arm-memory-tagging-extension.html.
  * Shacham (2007) Hovav Shacham. 2007\.  The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In _ACM Conference on Computer and Communications Security_. ACM, 552–561.
  * SiFive Inc. (2021a) SiFive Inc. 2021a.  SiFive FU540-C000 Manual (v1p4).  https://www.sifive.com/boards/hifive-unleashed.
  * SiFive Inc. (2021b) SiFive Inc. 2021b.  SiFive OpenEmbedded Layer.  https://github.com/sifive/meta-sifive.
  * Sinha et al. (2017) Kanad Sinha, Vasileios P. Kemerlis, and Simha Sethumadhavan. 2017\.  Reviving instruction set randomization. In _IEEE International Symposium on Hardware Oriented Security and Trust_. 21–28.
  * Snow et al. (2013) Kevin Z. Snow, Fabian Monrose, Lucas Davi, Alexandra Dmitrienko, Christopher Liebchen, and Ahmad-Reza Sadeghi. 2013.  Just-in-time code reuse: On the effectiveness of fine-grained address space layout randomization. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 574–588.
  * Song et al. (2016) Chengyu Song, Hyungon Moon, Monjur Alam, Insu Yun, Byoungyoung Lee, Taesoo Kim, Wenke Lee, and Yunheung Paek. 2016\.  HDFI: Hardware-assisted data-flow isolation. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 1–17.
  * Song et al. (2021) Wei Song, Boya Li, Zihan Xue, Zhenzhen Li, Wenhao Wang, and Peng Liu. 2021\.  Randomized last-level caches are still vulnerable to cache side-channel attacks! But we can fix it. In _IEEE Symposium on Security and Privacy_.
  * Spafford (1989) Eugene H. Spafford. 1989\.  The internet worm program: An analysis.  _ACM SIGCOMM Computer Communication Review_ 19, 1 (Jan. 1989), 17–57.
  * Szekeres et al. (2013) Laszlo Szekeres, Mathias Payer, Tao Wei, and Dawn Song. 2013\.  SoK: Eternal war in memory. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 48–62.
  * the PaX Team (2001) the PaX Team. 2001\.  Address space layout randomization.  https://pax.grsecurity.net/docs/aslr.txt.
  * Tice et al. (2014) Caroline Tice, Tom Roeder, Peter Collingbourne, Stephen Checkoway, Úlfar Erlingsson, Luis Lozano, and Geoff Pike. 2014\.  Enforcing forward-edge control-flow integrity in GCC & LLVM. In _USENIX Security Symposium_. USENIX Association, 941–955.
  * Tran et al. (2011) Minh Tran, Mark Etheridge, Tyler K. Bletsch, Xuxian Jiang, Vincent W. Freeh, and Peng Ning. 2011.  On the expressiveness of return-into-libc attacks. In _International Symposium on Recent Advances in Intrusion Detection_. Springer, 121–141.
  * Van Bulck et al. (2018) Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and Raoul Strackx. 2018\.  Foreshadow: Extracting the keys to the Intel SGX kingdom with transient out-of-order execution. In _USENIX Security Symposium_. USENIX Association, 991–1008.
  * Van Bulck et al. (2020) Jo Van Bulck, Daniel Moghimi, Michael Schwarz, Moritz Lipp, Marina Minkin, Daniel Genkin, Yuval Yarom, Berk Sunar, Daniel Gruss, and Frank Piessens. 2020\.  LVI: Hijacking transient execution through microarchitectural load value injection. In _IEEE Symposium on Security and Privacy_. IEEE, 54–72.
  * van der Veen et al. (2016) Victor van der Veen, Enes Göktas, Moritz Contag, Andre Pawlowski, Xi Chen, Sanjay Rawat, Herbert Bos, Thorsten Holz, Elias Athanasopoulos, and Cristiano Giuffrida. 2016\.  A tough call: Mitigating advanced code-reuse attacks at the binary level. In _IEEE Symposium on Security and Privacy_. IEEE Computer Society, 934–953.
  * Venkataramani et al. (2007) Guru Venkataramani, Brandyn Roemer, Yan Solihin, and Milos Prvulovic. 2007.  MemTracker: Efficient and programmable support for memory access monitoring and debugging. In _Provceedings of the International Conference on High-Performance Computer Architecture_. IEEE Computer Society, 273–284.
  * Waterman et al. (2021) Andrew Waterman, Tim Newsome, Chih-Min Chao, Yunsup Lee, Scott Beamer, and others. 2021\.  Spike RISC-V ISA simulator.  https://github.com/riscv/riscv-isa-sim.
  * Weicker (1984) Reinhold P. Weicker. 1984\.  Dhrystone: A synthetic systems programming benchmark.  _Commun. ACM_ 27, 10 (Oct. 1984), 1013–1030.