# SinClave: Hardware-assisted Singletons for TEEs

Franz Gregor  Scontain GmbHGermany ,  Robert Krahn  TU DresdenGermany ,  Do Le
Quoc  Huawei ResearchGermany and  Christof Fetzer  TU DresdenGermany

###### Abstract.

For trusted execution environments (TEEs), remote attestation permits
establishing trust in software executed on a remote host. It requires that the
measurement of a remote TEE is both complete and fresh: We need to measure all
aspects that might determine the behavior of an application, and this
measurement has to be reasonably fresh. Performing measurements only at the
start of a TEE simplifies the attestation but enables “reuse” attacks of
enclaves. We demonstrate how to perform such reuse attacks for different TEE
frameworks. We also show how to address this issue by enforcing freshness –
through the concept of a singleton enclave – and completeness of the
measurements. Completeness of measurements is not trivial since the secrets
provisioned to an enclave and the content of the filesystem can both affect
the behavior of the software, i.e., can be used to mount reuse attacks. We
present mechanisms to include measurements of these two components in the
remote attestation. Our evaluation based on real-world applications shows that
our approach incurs only negligible overhead ranging from $1.03$% to $13.2$%.

##  1\. Introduction

Trusted Execution Environment (TEE) technologies, such as Intel Software Guard
eXtensions (SGX), have gained much attention in the industry (Gordon, 2018;
Karnati, 2018) as well as in academia (Arnautov et al., 2016; Costan and
Devadas, 2016a; Tsai et al., 2017; Shen et al., 2020a). TEEs can protect
cloud-native applications (code and data) not only at rest, in transit, but
also during computation (in use) against powerful attackers with privileged
access to the underlying system software (e.g., the operating system or the
hypervisor), as well as, hardware. To ensure the confidentiality and integrity
of applications, TEEs execute their code and data inside an encrypted memory
region called _enclave_. Adversaries with privileged access cannot read or
interfere with the memory region and only the processor can decrypt and
execute the application inside an enclave.

Various hardware-assisted TEEs exist today, e.g., Intel SGX (Anati et al.,
2013; McKeen et al., 2013), ARM TrustZone (Winter, 2008), AMD SEV SNP (Sev-
Snp, 2020), and Intel TDX (Intel Corporation, 2020). Out of these TEEs, Intel
SGX has been widely adopted in practice (Gordon, 2018; Karnati, 2018) since
2015. It enables a general-purpose _ring 3_ -enabled TEE and yields a
significantly smaller Trusted Computing Base (TCB) in comparison with other
solutions where typically a whole Operating System (OS) is part of the TCB.
Furthermore, running legacy applications without source code changes is
supported by several SGX platforms (Arnautov et al., 2016; Priebe et al.,
2019; Tsai et al., 2017; Shen et al., 2020b).

Arguably, just executing legacy applications in Intel SGX environments is not
enough to ensure strong security guarantees. First, the legacy applications
were developed for a weaker threat model that trusts the underlying system
software. Thus, the SGX frameworks must employ additional protection
mechanisms to protect the naive legacy application against the elevated
threats in the SGX threat model. An example is the file system for which SGX
frameworks transparently encrypt file content before it is persisted by the
OS. The content’s integrity is additionally verified when files are read.
Second, deployment of legacy applications becomes significantly more
complicated in TEEs as configuration data (containing secrets) for these
applications must be hidden from potentially malicious system software.

To handle these problems, SGX frameworks, such as Graphene-SGX (Tsai et al.,
2017) or Occlum (Shen et al., 2020b), only provide the building blocks for
users and require individual implementation of secure data provisioning. This
approach contradicts the common goal to _support legacy applications_ without
changing their source code. Other frameworks, e.g., SCONE (Gregor et al.,
2020; Arnautov et al., 2016) and SGX-LKL (Priebe et al., 2019), provide an
integrated solution that transparently serves software configurations only
after it was verified that the TEE is genuine and that an adversary did not
alter the application.

The verification process is called _Remote Attestation_ (Costan and Devadas,
2016a) and allows one to establish trust in the application running inside an
enclave on a remote host. Remote attestation is the first step one performs to
ensure that an application on a remote host, runs the correct code inside a
genuine hardware enclave and it was initialized correctly. After that, users
establish a secure connection to the enclave to transfer secrets and
configuration to their application.

We show that current remote attestation mechanisms for TEEs, e.g., Intel SGX,
struggle to provide the expected trust and present an attack that bypasses the
remote attestation protocol. We further establish that the measurement during
attestation needs to be both complete, i.e., measure all aspects that
determine the behavior, and fresh, i.e., the measurement reflects the current
behavior. During runtime, applications may load libraries or open shared
volumes, which may change the behavior. The measurement should reflect the
dynamic behavior, and ensure the freshness (to protect against “reuse”
attacks), of the legacy applications running inside enclaves.

Unfortunately, performing measurements with current attestation mechanisms of
TEEs only ensures an application’s integrity at the starting of an enclave.
Thus, it’s vulnerable to “reuse” attacks, as we show in §3: one can use a
previous measurement to impersonate a TEE with a different expected behavior.
The vulnerability is based on the fact that almost all applications
dynamically load data, libraries, and configurations during runtime, which can
change their behavior. An attacker adds malicious code via libraries or
manipulates the configuration loaded into enclaves or loads data that might
trigger some application bug. They can then steal the secrets of an
application by bypassing the remote attestation as sketched in the following
example.

Imagine a service provider (i.e., cloud user) who wants to run a Python
application securely in a TEE in the cloud, e.g., an AI application based on
PyTorch or TensorFlow. The service provider triggers the application’s cloud
deployment and subsequently uses remote attestation to assure the TEE was
constructed without manipulations before securely sending the application’s
secrets. The attacker can observe and intercept this attestation process by
starting the requested Python interpreter correctly in the TEE but configuring
it such that it will load a malicious library to execute a TEE impersonator
program. This impersonator program then will correctly engage in the remote
attestation procedure with the service provider and obtain his secrets for the
Python application. The service provider cannot detect this manipulation since
the attestation procedure only captures the TEE’s initial state which the
attacker did not alter.

Vulnerabilities in a program’s configuration handling code – typically not
considered a vector of attack – can make any program exploitable to the
“reuse” attack. Considering this attack, Haven (Baumann et al., 2015) proposed
to incorporate configurations into the TEE such that the attestation would
also include them. However, this would also reveal the TEE’s configuration
and, thus, its secrets. An alternative solution integrates the service
provider’s cryptographic identity into the TEE and enforces that the
application would only start with correctly signed configurations. This
approach decreases the system’s usability as every TEE needs to be
individualized, prohibiting established procedures for software distribution
and deployment.

We overcome the portrayed issues by proposing a novel protection mechanism
against “reuse” attacks on remote attestation. Our solution does not break the
usability of legacy applications, e.g., supporting secure dynamic
configuration and secure mounting of volumes, and incurs only a negligible
overhead. Our contributions are as follows:

  * •

We provide a comprehensive description of remote attestation in TEEs,
especially Intel SGX (§2.2 and §3.1).

  * •

We illustrate a practical attack on TEE secret provisioning, subverting all
TEE protection goals (§3).

  * •

We highlight a previously neglected vector of attack for TEE software
rendering the majority of systems vulnerable to the presented attack (§3).

  * •

We show the novelty of our attack by exploiting state-of-the-art TEE runtime
systems (§3.2).

  * •

We implement and evaluate a new flexible secret provisioning mechanism
preventing the presented attack while supporting changes of input files,
arguments, environment files etc., as well as software updates (§4).

  * •

We conduct micro- and macro-benchmarks to demonstrate that our protection
mechanism incurs only negligible overhead and can be used in practice (§5).

##  2\. Background

###  2.1. Trusted Execution Environments

Trusted Execution Environments (TEEs) (Singh et al., 2021, 2020) such as Intel
SGX (Gordon, 2018; Costan and Devadas, 2016b), ARM TrustZone (Winter, 2008),
AMD SEV SNP (Sev-Snp, 2020), and Intel TDX (Intel Corporation, 2020), support
the design of complex, yet secure, applications. A TEE is a tamper-resistant
processing environment that guarantees the authenticity, integrity, and
confidentiality of its executing code, data, and runtime states (e.g., CPU
registers, memory, and sensitive I/O). The content of TEE remains resistant to
software attacks even from privileged code, as well as any physical attacks
performed on the main memory of the system. In addition, TEEs offer
_attestation_ for proving their trustworthiness to third parties. However, we
demonstrate the vulnerability of remote attestation by demonstrating a serious
“reuse” attack on the TEE of Intel SGX (§3). We also introduce a mechanism to
protect against this attack in practice (see §4).

###  2.2. Intel SGX Security Argument

Before introducing the attack that bypasses remote attestation, we provide a
detailed description of the technical building blocks that our work builds
upon. These are primarily the SGX enclave creation, measurement, and
attestation procedures. Enclave measurement and attestation are implemented
mostly in hardware, such that they cannot be modified by a user. Therefore, we
have to rely on existing functionality to establish a practical solution that
is usable with current hardware. In the following, we will explain the
fundamentals of the security argument for attestation in SGX and show how
existing frameworks use SGX attestation and the resulting ramifications. We
further introduce the concepts for validating that an enclave has not been
tampered with before its initialization. We omit aspects of SGX that are
superfluous for the understanding of this work and that have been sufficiently
explained elsewhere (Costan and Devadas, 2016a).

_Enclave verification_ begins with the creation of the enclave itself:
Representations of each executed operation are hashed into a measurement
value. Enclave creation is finalized with enclave initialization during which
signer-based verification properties are bound to the enclave. Once
initialized an enclave can engage in remote attestation to convince the
verifier of its integrity.

Figure 1. Enclave Creation Procedure.

No description.

2.2.1 Enclave Creation

An SGX enclave is created by a program referred to as the starter, which is
not part of the TCB (Intel, 2020). The starter utilizes four SGX instructions
to create the enclave as shown in Figure 1. ECREATE creates an enclave, more
precisely, its metadata SGX Enclave Control Structure (SECS) and, therefore,
determines its maximal size and enclave attributes, such as 64-bit mode, debug
mode, or whether it requires certain extended processor features such as
Advanced Vector Extensions (AVX) or Control-flow Enforcement Technology (CET).
Once the enclave is created, EADD and EEXTEND instructions are used to add
each page (4 kB) of code and data to the enclave’s memory and extend its
measurement respectively. Once the enclave’s memory is set up, the enclave is
initialized with EINIT preventing any further modifications of the enclave
from the outside. In the initialized state the enclave can be entered and
assumes control over itself.

Figure 2. Schema of SHA256.

No description.

Our solution tightly interacts with the _hash algorithm_ used for enclave
measurement. Hence, we provide the necessary detail about the utilized SHA-256
algorithm. SHA-256 (Penard and van Werkhoven, 2008; PUB, 2012) is one variant
of the Secure Hash Algorithm 2 cryptographic hash functions. The SHA hash
functions are based on Merkle-Damgård construction (Merkle, 1979) with Davies-
Meyer compression functions. SHA-256 maintains an internal state of 256-bits
that are organized in 8 values ($H1$ …$H8$) with 32-bit each that eventually
becomes the hash value. An arbitrary message, whose length is only limited by
the hash internal length counter of 64-bit, is compressed into these 8 values.
Calculating a SHA-256 hash value can be summarized in three steps (see Figure
2):

_Initialization \raisebox{-.9pt} {1}⃝_: The hash state is initialized with
initialization vector values according to the SHA-256 standard.

_Updating \raisebox{-.9pt} {2}⃝_: Each 64-byte chunk of message data is
processed iteratively with a procedure, we summarize as SHA Round and by
updating the length counter. The SHA Round consumes the incoming message chunk
and intermingles it with the current hash state ($H1$…$H8$). The output of the
SHA Round becomes the new hash state.

_Finalization \raisebox{-.9pt} {3}⃝_: After the entire message is consumed
through iterative updating, the hash calculation is finalized yielding the
hash value. For this, the message length counter value is hashed into the hash
state using the SHA Round procedure. The 64-bit message length is padded
according to a standardized padding scheme to fill the entire 64-byte input of
the SHA Round. This final result of the SHA Round calculation is the SHA-256
hash value of the input message.

Note that, due to the construction of the hash algorithm, for every $64$ Bytes
of compressed input an internal intermediary state is reached that can be
described with $256$ bit of internal hash state and $64$ bit of already
compressed input. During an SGX enclave measurement, each operation’s
representation is a multiple of $64$ Bytes in length. It follows that, after
each enclave construction operation the enclave measurement hash calculation
is in a state that can be described as said before.

2.2.2 Enclave Initialization

Before locking the enclave from further modifications, e.g., from system
software, the enclave initialization verifies the Enclave Signature Structure
(SigStruct). The SigStruct is created by the enclave signer, e.g., the
software owner, and specifies the expected enclave measurement value
(MRENCLAVE), enclave attributes, the signer assigned product id, and a
security version number. The information is protected by an RSA signature
created with the enclave signer’s key. Upon enclave initialization, the
processor will ensure that the SigStruct has not been manipulated and that the
constructed enclave matches it. To summarize the SigStruct enables the signer
to specify the SGX configuration for a particular enclave and to specify
access to specific SGX sealing keys.

The SGX SigStruct can not prevent manipulations entirely. As it is handled by
system software, the adversary is free to modify it and subsequently sign it
with their own key.111An adversary may execute an enclave with a configuration
not intended by the signer to allow for exploitation. Therefore, the signer’s
identity (MRSIGNER) is incorporated during the attestation procedure (detailed
next).

Another mechanism utilized during enclave initialization is the verification
of the EINIT Token (EINITTOKEN) data structure. An EINITTOKEN allows an
enclave with a certain MRENCLAVE measurement, MRSIGNER identity, and enclave
attributes to start. It is generated by a dedicated system enclave, the launch
enclave, and must be provided to the EINIT instruction initializing the
enclave for it to succeed. In the first implementation of SGX, only enclaves
signed by Intel were allowed to become launch enclaves. The launch enclave
from Intel generated EINITTOKEN s for any enclave in debug mode. However,
enclaves in production mode must be signed with a signer’s key that is
authorized by Intel to obtain a EINITTOKEN data structure. Later Intel
introduced Flexible Launch Control (FLC) allowing the platform owner to
determine which launch enclaves can run. With FLC the default launch enclave
allows any enclave to run in production mode.

2.2.3 Attestation with SGX

When an enclave is running (initialized and entered) it can request a report
from the processor. The report entails the enclave’s MRENCLAVE, MRSIGNER,
enclave attributes as well as information about the platform’s processor. With
the help of a quoting enclave, the report can be transformed into a remotely
verifiable quote. Note that the report and quote show the enclaves’ state at
the point of initialization. That is, any modifications afterward (by the
enclave) are not reflected in the quote and are, thus, not observable by the
remote verifier.

Intel has defined protocols for local and remote attestation to be used for
verifying the genuineness of an SGX enclave and the software executed within
the SGX enclave (Anati et al., 2013). SGX’s attestation scheme relies on a key
generation facility and on a provisioning service. During the manufacturing
process, a “Seal Key” and a “Provisioning Key” are burned into the die of an
SGX-enabled processor. The provisioning secret is generated by Intel at the
key generation facility and also stored by the provisioning service for later
identification. The seal key is generated by the processor and not known to
Intel (Costan and Devadas, 2016b).

Intel distinguishes between local and remote attestation. Both describe a
process of an enclave proving its identity to a second dedicated enclave. As a
result, two enclaves on the same platform have established trust and can
exchange information.

During attestation a dedicated instruction (_EREPORT_) defined by the SGX-SDK
is used to create a report that cryptographically bundles, most importantly, a
measurement of the enclave (_MRENCLAVE_), a certificate (identity information)
of the enclave’s software (_MRSIGNER_) and a hardware MAC to prove
authenticity. Other information such as the CPU’s security version number is
included as well but herein not further explained. The attestation report can
then be used to verify that it was produced by identified software isolated by
genuine (Intel-) hardware.

Remote attestation is executed after the local attestation has been executed
successfully and a local quoting enclave has established trust in the to-be-
attested enclave. Through remote attestation, a third party (challenger) can
remotely establish trust in an SGX-enclave and ensure the genuineness of the
executed soft- and hardware. Section 3.1 explains attestation in more detail.

###  2.3. System & Threat Model

All systems building upon SGX must rely on its hardware-assisted attestation
mechanism to establish trust. In SGX’ threat model the processor is the only
trusted component. In this section, we assume an attacker who is not able to
violate the security guarantees of SGX. However, attackers can create enclaves
in the same environment as the user to impersonate the user’s enclave and
create attestation reports with arbitrary reportdata content.

Our system model has the following assumptions:

  * •

The software is developed and distributed by a trusted software developer.

  * •

A trusted signer prepares the software for execution inside of SGX. This is
done with the help of a trusted SGX framework that enables the execution of
legacy software inside of SGX. In particular, the signer creates a valid
SigStruct.

  * •

The user customizes the SGX-enabled software by configuring it. The
configuration (files, environment variables, program arguments, etc) is
secured through trusted means provided by the SGX framework.

  * •

Developer, signer and user and their systems are benign and have means to
securely exchange information about the integrity of the software.

  * •

The customized software is transmitted and executed in an SGX-enabled
environment under control of the adversary.

  * •

The utilized SGX-environment delivers its security goals. For example, the
remote attestation mechanism and infrastructure are trusted and we assume SGX
processor implementation is not exploitable.

  * •

The attestation protocols are known to the adversary, i.e., no security
through obstruction.

Note that the developer, signer and/or user roles might be shared by the same
entity. SGX cannot protect secrets at rest for previously unknown platforms.
Therefore, secrets can not be part of an enclave but must be provisioned after
the enclave has been found trustworthy via remote attestation.

In our system model, the user wants to run a native application in an enclave
in a public cloud. The enclave needs a configuration to run and secrets to,
e.g., authenticate to other services or decrypt sealed file system content.
The user utilizes SGX remote attestation to build trust into the remote
enclave before sending its configuration via a secure channel. The secure
channel is bound to the enclave via attestation. The attestation and
configuration tasks are delegated to a trusted verifier service on behalf of
the user.

Note that our system model is closely modeled after what we see in existing
systems: For example, SGX-LKL enclaves load an encrypted disk image and set
program arguments and environment variables according to the configuration
(Priebe et al., 2019). The trusted verifier service is the sgx-lkl-ctl command
line tool that attests the enclave and delivers the configuration. The secure
channel is based on wireguard and it is bound to the enclave by adding the
enclave’s public wireguard key in its reportdata field. In a SCONE, program
arguments, environment variables, and session secrets are retrieved from the
configuration and encrypted file system content is opened before the internal
program is loaded. The SCONE Configuration and Attestation Service (CAS)
(Gregor et al., 2020) is the trusted verifier service to which the enclave
connects and from which it retrieves its configuration through encrypted
channel. Recently, remote attestation was added to Graphene-SGX (Tsai et al.,
2017) as a shared library that uses RA-TLS as a secure channel bound to the
enclave.

##  3\. Remote Attestation Bypass Attack

In this section, we describe the remote attestation bypass attack in which the
adversary uses an enclave trusted by the user to gain the user’s secrets.
First, we provide an overview of remote attestation in TEEs, then we show the
vulnerability of the general remote attestation protocol provided in TEEs.
Afterward, we describe the attack against remote attestation in Intel SGX.

Figure 3. General remote attestation protocol in TEEs.

No description.

Overview of remote attestation attack: If the application dynamically loads
libraries and its configuration during runtime, an attacker can load
compromised libraries or manipulate the configuration of enclaves.

Figure 4. The overview of remote attestation attack.

###  3.1. Attestation Overview

Remote attestation provides the ability for a verifier to assert a certain
state of an application is running in a TEE (see Figure 4). On the device,
there is a prover entity (_Quoting Enclave_ in SGX) able to produce the
evidence for the state of the application before running it; and securely send
it to the verifier. The attestation process consists of two major parts: _(i)_
Characterization of the application’s current state (the measurement process
to create the evidence, called _quote_ in SGX); and _(ii)_ Secure transfer of
the evidence to the verifier.

The measurement is used to prove the application is running in a correct state
(e.g., correct code) inside enclaves of TEEs. It can be a cryptographic hash
over the binary of an application (see §2) or could also contain other target
information for the attestation. The measurement is then signed using a
hardware-protected certified key unique to the device to create the evidence
or quote that can be verified by the verifier. The certified key ensures the
_authenticity_ of the remote attestation protocol. To ensure the _freshness_
of the evidence, the verifier includes a _nonce_ into the challenge in its
request to the remote enclave.

Figure 4 summarizes the general remote attestation workflow in TEEs. Before
launching an application in an enclave deployed on the remote device/host, the
measurement process creates the attestation evidence. When the verifier wants
to verify if the remote application has an expected state, it sends a request
with a nonce to the remote enclave (step (1)). The enclave combines the nonce
with the measurement, signs it to create a report using attestation keys
(stored in CPU hardware), and sends it to the prover (step (2)). The prover
verifies the report, signs it to create the evidence (i.e., quote), and sends
it back to the verifier (step (3) and (4)). The verifier checks the signature
and certificate with the help of the trusted third-party, e.g., the
attestation service of the TEE provider, and checks that the measurement
indicates the start of the expected enclave (steps (5) and (6)). Note that in
other TEEs, e.g., AMD SEV SNP, the enclave is a secure virtual machine (VM)
and the remote attestation and quoting are performed by the AMD Secure
(Co-)Processor.

###  3.2. Remote Attestation Bypass Attack Overview

A significant security issue in remote attestation of current TEEs is that the
measurement process is only conducted before running the application and
cannot be performed during runtime. If the application dynamically loads
libraries and its configuration during runtime, an attacker may load
compromised libraries or manip- ulated configurations into enclaves (see
Figure 4).

The attacker can create a genuine evidence (quote) using the malicious enclave
since it has correct measurement and send this evidence to the verifier to
bypass the remote attestation (steps (2), (3), and (4)). After bypassing the
remote attestation, the attacker can steal secrets (details in §3.3).
Applications with the pre-compiled binaries (e.g., Python) are vulnerable to
this attack since their configuration mainly determines their behavior.

In this work, we imagine a user who wants to run their legacy application in a
potentially adversarial cloud environment. The user protects the application’s
code and data during runtime using an SGX framework. At runtime, the
application’s behavior significantly depends on data the application consumes
(mainly from the file system), e.g., configuration files or mounted volumes.
Remote attestation is subsequently used to establish that the application has
not been modified by an adversary prior to execution.

The adversary needs two components for the attack: an TEE impersonator and a
report server. The TEE impersonator implements the attestation protocol
utilized by the verifier, while the report server generates SGX reports with
arbitrary, adversary chosen reportdata fields. Note that the TEE impersonator
is not required to be executed inside of an enclave. The report server, on the
other hand, must be executed inside of an enclave, fulfilling expectations by
the verifier and allowing it to generate SGX reports. Next, we describe how
the adversary can build a report server from an enclave without this being
reflected in the enclave’s measurement.

Creating a Report Server. A report server is an enclave that produces SGX
reports with arbitrary report data upon the adversary’s instruction. The
concrete steps and complexity of creating a report server from an arbitrary
enclave varies but three operations must be implemented:

  * •

The desired report data and target information must be copied into enclave
memory.

  * •

The SGX report generation must be invocable.

  * •

The created report must be delivered to a verifier.

Network connections, file system interfaces, the terminal, or reading of
environment variables can be utilized for communication. Arguably any enclave
must provide some form of input and output to do useful work. We also assume
that any enclave of interest would allow for SGX report generation given that
SGX attestation is to be used to securely provision configurations to the
enclave and that remote attestation can be used throughout the lifetime of an
enclave. For example, attestation is used to convince users of a service of
its trustworthiness or sign data produced by the enclave. Many SGX frameworks
voluntarily expose attestation features including setting arbitrary
reportdata: SCONE (Arnautov et al., 2016) exposes report generation via C
functions to user code. Occlum (Shen et al., 2020a) provides special ioctl
system calls. Gramine (Corp, [n. d.]) (initially Graphene-SGX (Tsai et al.,
2017)) exposes the report generation via pseudo-files in /dev/attestation.
Hence, we assume that the three necessary operations are present in any
enclave an adversary would be interested in.

With these building blocks available, the adversary needs to gain control over
the enclave such that it uses these operations according to the adversary’s
desire. Such control is obtained by exploiting vulnerabilities in the
enclave’s code and, thus, many frameworks use memory-safe languages to
mitigate such risks. However, we found a significantly simpler approach:
through configuration.

Creating a Report Server by Configuration. Our threat model allows adversaries
to use significantly wider attack vectors than in a more traditional model. In
particular, they can start and configure the user’s enclave arbitrarily often.
They can also run their own verification and configuration component that
delivers the desired configuration to the enclave. In summary, if not
prohibited by the enclave itself, the adversary can run the user enclave with
an arbitrary configuration.

If the software’s configuration that is not reflected in the enclave
measurement (§ 3.3) dictates the to-be-loaded program, the adversary can
simply start a report server implementation. In other cases, the user
application might be substantially configurable. This is mostly always the
case for language interpreters, for languages such as Java, Python,
JavaScript, Ruby, etc. In some frameworks, these interpreters are already the
entire enclave while the actual program is loaded from the file system at
runtime. For example, when Java applications, such as Wildfly/JBoss or Tomcat,
are executed with the exact same interpreter and, thus, the enclave is used.
Thus, the adversary can configure the interpreter enclave to load their report
server implementation in the necessary language, for example, Java without the
attestation report reflecting this.

Alternatively, the adversary can use any program that loads dynamic code
during runtime. Dynamic loading is broadly used to load optional features into
a program. For example, Apache httpd loads only the configured modules, or
NGINX’s dynamic module extends its features using dynamically loaded code.
Since the dynamic code is only loaded during runtime, it is typically not
reflected in the enclave measurement and, thus, manipulating it is not
revealed through attestation. Therefore, the adversary can patch existing
functions such that they execute the desired operations. This code could read
the desired reportdata from a file, query the SGX report, and write it into
another file implementing a report server. With this, the adversary could
trigger the code by querying a site with the appropriate compression algorithm
requested.

A straightforward way to handle the attacks is using static linking for
applications. However, it also comes with the cost of increased binary size
which makes booting time slow, compatibility issues, performance overhead, and
resource redundancy. For interpreter languages, e.g., Python they need to pack
dependencies with interpreter engines which is cumbersome. It requires heavy
engineering, especially with complex libraries such as TensorFlow.

Besides SGX, other TEEs, such as AMD SEV and Intel TDX, have a similar issue
since these TEEs only perform measurements and ensure the integrity of
confidential VMs after the booting process. During computation, these TEEs do
not perform any measurements if shared data or libraries from the host are
loaded to the VMs; it requires carefully checking or adding their measurement
during the boot process. If there is no shared data or libraries between VMs
or with the host, an attacker can just boot the VM from a victim and perform,
e.g., side-channel attacks on the VM in a lab environment.

###  3.3. Attack Examples

In this section, we show the practicability of the previously described attack
by describing our attack against state-of-the-art SGX frameworks. Compared to
previous work (van Schaik et al., 2020), our approach does not attack the
sealing mechanism of TEEs. We show our attack against applications running in
SCONE as well as in SGX-LKL which support transparent remote attestation for
users.

3.3.1 Attacking Python and NodeJS in SCONE

We assume the Python application’s files are protected with SCONE’s file
system encryption. The application’s configuration, including the Python
interpreter’s expected enclave measurement and signer identity and the file
system encryption key, are stored in SCONE’s CAS (Gregor et al., 2020) the
secrets management service (running in an enclave itself). SCONE enclaves
connect to CAS during startup to obtain their configurations and transparently
provide it to the executed application. For this, CAS ensures that the
enclaves match the user-provided expectations, utilizing SGX attestation. That
is, in SCONE, CAS functions as the attestation verifier.

TEE Impersonator. For building a TEE impersonator for SCONE we adapted the
existing CAS-client implementation. The SGX report retrieval mechanism of the
client is configurable through a callback mechanism. Hence, we only needed to
implement a client for the SCONE report server implementation (discussed
below) and configure the CAS client to use it. Together with the necessary
argument parsing the TEE impersonator implementation consists only of $75$
lines of code.

Report Server. In SCONE, a program’s file system is not reflected in the
enclave measurement value. Any Python program utilizing the same Python
interpreter in SCONE uses an identical enclave and can not be distinguished by
the verifier. The verification of the actual Python (byte-) code that is
stored in the encrypted file system is delegated to the runtime. With the
configuration, the runtime obtains the expected state of the file system and
uses this information to ensure files have not been tampered with during
execution. This delegation is precisely the exploitable culprit as the passed
configuration is not verifiable through attestation. Based on this, an
adversary can invoke the user’s Python enclave and configure it to run a
report server program instead of the user’s application (after attestation).

For the attack, we implemented a report server with a simple Python socket
server that receives the information necessary to query the report, in
particular the reportdata, via the network. The SGX report is obtained through
a custom Python C module in which we use the EREPORT SGX hardware instruction
in assembly. The resulting report is sent via the same network connection
towards the TEE impersonator. Our attack requires dynamic loading of code at
runtime as required by the Python interpreter. The entire report server
implementation consists of $33$ lines of Python code and $76$ lines of C code.

For the attack on NodeJS, we implemented a separate report server as the
interpreter binaries. NodeJS allows the execution of native (C) code through
C++ addons that are loaded dynamically at runtime. We implemented the server
loop in $22$ lines of JavaScript and the addons executing EREPORT required
$67$ lines of C++ code.

Attack Procedure. For the attack, the adversary first starts the report server
built from the enclave trusted by the user. Then the adversary invokes the TEE
impersonator configuring it with the user’s configuration ID as well as the
addresses of the necessary services and waits for the TEE impersonator to
yield the user’s configuration.

The TEE impersonator connects to CAS and requests the user enclave’s
configuration. The connection is established via a secure TLS channel whose
certificate key’s are used to ensure the authenticity of the connection, i.e.,
ensuring the channels integrity but also that the channel is terminated in the
expected enclave. In response CAS will demand a suitable remote attestation
quote (SCONE, EPID, or DCAP quotes are supported). Depending on the system
capabilities an appropriate mechanism is chosen by the impersonator and the
necessary SGX report is queried from the report server. The report server is
instructed to incorporate the TEE impersonator’s certificate key into the
report’s reportdata field undermining the channel’s authenticity. The verifier
(CAS) is unable to detect the manipulation because the quote is valid, it
shows the expected enclave is running and the TLS channel utilizes the
expected certificate whose key is incorporated into the report. Thus, it
grants access and delivers the requested configuration.

3.3.2 Attacking Python in SGX-LKL

In SGX-LKL, the enclave starts a simple attestation and configuration service
when invoked with sgx-lkl-run. The user then connects to that service using
the sgx-lkl-ctl tool and attests the enclave. For this, the tool shows the SGX
report/quote details on the screen. After verification of the shown details,
the user invokes sgx-lkl-ctl again to transmit the enclave configuration,
which contains the disk encryption key. SGX-LKL uses encrypted disk images to
load code into the enclave. Having received the configuration, the enclave
runtime accesses the disk and invokes the user application.

SGX-LKL implements additional restrictions to ensure the integrity of the
attestation procedure. For example, the runtime enforces that attestation and
configuration is only done once. The integrity of the attestation and
configuration procedure is enforced by a public wireguard key that is provided
to the enclave during invocation. That is, the enclave will only interact with
attestation and configuration requests originating from the benign user.
Furthermore, we assume the system is configured to run in release mode, which
adds additional restrictions enforced by SGX-LKL: The disk image (program)
must be encrypted, and the SGX-LKL will not start the user program without
attestation. We will show beneath that only this last restriction influences
the attack but does not mitigate it.

TEE Impersonator. SGX-LKL’s attestation and configuration interface is
implemented in C and uses _protobuf_ for message framing. We implemented the
SGX-LKL impersonator in Python. It opens the attestation and configuration
ports and waits for the user to connect. SGX-LKL utilizes Wireguard (VPN) to
establish secure channels. We instruct the host OS to provide the VPN expected
by the client instead of handling these details in the impersonator. Upon
start, the impersonator is configured with the nonce expected by the user and
the public wireguard key the adversary configured the VPN with. Our
implementation consists of $236$ lines of code and about $2000$ lines of
generated _protobuf_ Python code.

Report Server. The fact that SGX-LKL uses encrypted disk images and that the
enclave is attested before the user code can be loaded – since it is encrypted
– hints that attestation in SGX-LKL ensures the integrity of the framework but
not the user application. User application integrity verification is
implemented by the runtime by verifying the disk image’s integrity. In
consequence, two different programs running in SGX-LKL will, from SGX
attestation perspective, be the same as long as they use the same version of
SGX-LKL. Consequently, the verifier can not distinguish them posing little
restrictions for our report server implementation. We implemented it in $152$
lines of C code.

Attack Procedure. The attack on SGX-LKL requires the adversary to interfere
with the commands sent by the user. The adversary is in control over the
machine on which the user wants to execute the enclave which allows them to
execute all necessary actions. Before the actual attack, the adversary
prepares an encrypted disk image containing the report server. When the user
attempts to start their enclave with sgx-lkl-run, she provides their public
wireguard key and a nonce (and the desired disk image) to the command to
ensure the enclave can only be controlled by them. The invocation command is
intercepted by the adversary and they start the report server’s disk image,
however, with the adversaries public wireguard key using the sgx-lkl-run
command. This enclave is then attested and configured by the adversary to
execute the report server. At the same time, the adversary starts the SGX-LKL
impersonator and configures it to put a public wireguard key controlled by the
adversary into the SGX report. Incoming requests to the user’s enclave are
forwarded the configuration interface of the SGX-LKL impersonator using the
same wireguard guard key. Afterward, the adversary waits for the attestation
request from the user that is forwarded to the SGX-LKL impersonator. It
queries the report server for the necessary SGX report containing the
adversary’s wireguard key in the reportdata field. The report is sent back to
the user for inspection. At this point the user has no chance to detect the
manipulation, seeing only a valid SGX report for the expected SGX-LKL enclave.
Thus, they decide to trust it and use the embedded wireguard key to establish
a secure connection to send the configuration over. The wireguard key is
embedded in the report and the connection is established with the wireguard
setup in the malicious host OS responded to by the impersonator, revealing the
configuration to the adversary.

##  4\. Attestation Protection Mechanism

The remote attestation bypass attack uses fundamental technical properties
built into SGX rendering a verifier unable to distinguish between a newly
started enclave without a configuration and an enclave pre-configured by an
adversary. In the following, we detail our protection mechanism against the
remote attestation bypass attack.

###  4.1. Design Space

Different approaches for protection mechanisms are possible due to the complex
design space of remote attestation and the software stack involved. However,
we adhere to two common characteristics of modern cloud-native applications:
An enclave’s configuration might not be known beforehand, i.e., program
arguments of spawned sub-processes might be computed dynamically by the parent
process and files can be used to transfer data between processes. Hence, the
protection must be _flexible_ to adapt to the dynamic behavior of applications
and the protection mechanism must be practical, i.e., usable with existing
hardware.

###  4.2. General Solution Principle

We prevent the shown exploitation by enabling the verifier to establish that
an enclave is fresh and complete, i.e., it cannot have received a malicious
configuration beforehand. One possibility to prevent the attack would be to
forestall the adversary configuring the enclave into a report server. However,
preventing the adversary from freely configuring the enclave would also
contradict our _flexibility_ requirement. Next, we discuss solutions with
specific shortcomings before presenting our solution – _Singleton Enclaves_
(§4.4).

###  4.3. Naive Approaches

Restricted Runtime Operations. A naive approach implemented in SGX-LKL allows
the enclave to run only if it is attested and its configuration is provisioned
via a secure channel that must be terminated by the trusted verifier. Still,
we can exploit it (as shown before) because the trusted verifier must also be
configured in the enclave, and this configuration is not reflected in the
attestation report and, thus, can not be verified. Therefore, the adversary
can start an enclave, configure it to trust the adversary, and start a report
server. This report server can then be used to fool the user by pretending to
be a fresh SGX-LKL enclave since its SGX report neither reflects that it was
configured already nor that its actually SGX-LKL runtime was configured to
trust the adversary.

Using SigStruct. As an alternative to the previous approach, the SGX SigStruct
data structure could store additional information without the need for
adaption of the software. However, there are only very limited fields/space
that can be freely controlled – a couple of bytes. Unfortunately, this is not
enough to store a cryptographic identity but merely a config identifier. Using
config identifiers is insufficient as their enforcement would have to be
implemented in the verifier that is suspectable to manipulation by the
adversary. Imagine a SigStruct stating that the enclave only accepts
configurations with id A. The adversary could simply accept that SigStruct and
ignore the restriction. Alternatively, a hash of an entire config would be
equivalent to the comprehensive enclave measurement solution but unfortunately
would be too large for the data structure.

The _MRSIGNER_ field could also be used for the protection: The enclave could
only accept configuration that comes from a specific signer identity. This
approach would work, however, it requires the users have to sign all
individual enclaves themselves. We regard this approach as impractical for
cloud-native applications.

Using reportdata to show state. As an alternative solution, one could include
a config identifier or similar that proves the enclave was not configured
already into the report data field of the report. After all, the report must
be created from the code that was measured in it and thus only that code could
say it has not been configured at runtime. This approach has been supported by
SCONE and SGX-LKL but using the report data field for such purposes is not
secure in our threat model. After an adversary manages to create a report
oracle, she also controls the report data. Consequentially, only fields set by
the hardware can be trusted against a powerful adversary.

###  4.4. Singleton Enclaves

Figure 5. Enclave Memory Layout in SinClave

The Figure shows three memory layouts for enclaves. It is presented how
SinClave adds a single page to the enclave memory.

Our proposed protection mechanism is based on the following observations. The
attestation token is a mechanism that individualizes an enclave’s measurement
value (MRENCLAVE) in a verifiable manner such that re-attestation can be
detected. This individualization happens at enclave construction time and is
thus compatible with the binary distribution of software.

The basic idea behind our solution is that system software adds an additional
page, the _instance page_ (see Figure 5), to the enclave dynamically during
enclave construction. This page, in particular, contains an attestation token,
and the verifier’s cryptographic identity. The attestation token is unique and
generated by the verifier in a previous step.

Once the enclave is started it directly engages in attestation to obtain its
configuration from the verifier. In the request, the enclave sends its SGX
quote and the attestation token to the verifier via a secure channel. The
verifier calculates the expected MRENCLAVE based on the expected enclave, the
attestation token, and its own cryptographic identity and can thus, despite
individualization, verify the enclave’s integrity.

The enclave utilizes the verifier identity that is part of the _instance page_
to ensure it only obtains configuration for the intended verifier. This
prevents an adversary from supplying its malicious configuration and the
verifier can utilize the attestation token to keep track of attested enclaves
ensuring each enclave is attested exactly once. In the following, we discuss
the most important aspects of our approach.

Verifiable Enclave Extension. Adding an additional page to the enclave must
necessarily yield the desired different enclave measurement. On the other
hand, the verifier must be able to ensure the intended enclave was extended
correctly, i.e., it was neither manipulated beforehand nor incorrectly
extended. Adding an additional page to the end of an enclave requires the
additional execution of one EADD and four EEXTEND SGX instructions. From the
perspective of the enclave measurement SHA-256 calculation, adding an
additional page to an enclave constitutes four additional update calls. This
allows the verifier to pre-calculated the expected enclave measurement: The
SHA-256 calculation’s internal state needs to be instantiated just before the
initial (not yet individualized) enclave was finalized with EINIT and updated
according to the expected content and position of the _instance page_.
Position and content of the instance page, as well as, the necessary SHA-256
calculation state can be computed by the verifier given the enclave.

On-Demand SigStruct Creation. The SigStruct is necessary to initialize an
enclave (§ 2.2) but is only valid for a specific enclave measurement value
(MRENCLAVE). Since the attestation token individualizes enclave’s MRENCLAVE, a
suitable SigStruct must be computed for each individualized enclave before it
can be initialized. Computing the SigStruct requires knowledge of the enclave
signer’s private key. This key must stay private: (a) If the adversary knows
the key, we could only use the enclave measurement for verification. (b) Any
enclave with the signer identity A can derive signer identity A’s SGX sealing
keys. If the adversary could sign his own enclaves, the SGX sealing keys would
be compromised. (c) In SGX v1 only (signer) identities whitelisted by Intel
can run enclaves in production mode. Compromised keys would be blacklisted.

In our protection mechanism, we distinguish _singleton enclave_ and _common
enclave_. Both have the same base enclave hash – the same software/code.
However, the singleton enclave must be attested and have a unique _MRENCLAVE_
while the common enclave can be started arbitrarily many times. The _instance
page_ of the common enclave is zeroed such that the runtime can decide whether
it requires attestation or not.

When querying the verifier for the attestation token, the common enclave’s
SigStruct is also sent to the verifier. The verifier ensures it matches the
base enclave hash (if instantiated for the common enclave) and if so finalizes
the base enclave hash according to the attestation token it just chose. The
verifier uses that value to create the (on-demand) SigStruct for the
particular singleton enclave he expects to be started according to the base
enclave hash, the token and his certificate. The SigStruct is equivalent to
the common enclave’s SigStruct except that the enclave measurement and the
signature is exchanged. That SigStruct is sent to the starter who will
initialize the enclave with the instance/singleton page and the received on-
demand SigStruct. This ensures that the signer key never leaves the trusted
verifier and only enclaves the signer previously signed already (he trusted)
will be attested. In consequence, our protection satisfies the requirements
described in §4.1.

Note that the goal of the proposed protection mechanism is to allow users to
create a singleton SGX enclave that can be extended to a singleton
confidential VM. In the singleton confidential VM, the measurement process
during booting time includes the user identity and security for that VM. In
other words, an attacker cannot just boot a VM of the user since he/she cannot
generate the unique measurement of the VM. Only after verifying that the VM is
a unique single instance, secrets are injected into the VM.

##  5\. Evaluation

This section presents the experimental evaluation of SinClave using micro- and
macro-benchmarks. In the evaluation, we focus on the system behavior expected
to be influenced by our contributions. These are enclave signing, enclave
construction and attestation. During the signing SinClave’s interruptible
SHA-256 implementation is used. We expect it to impose a lightweight overhead
since it is implemented using only Rust, while state-of-the-art SHA-256
implementations use optimized assembly code and hardware acceleration (Gulley
et al., 2013). An enclave based on SinClave must query the verifier before the
singleton enclave can be initialized. This communication is necessary to get
the attestation token and the on-demand SigStruct and will slow down the
overall application start. Finally, the verification with SinClave requires
additional checks, which we expect to have only negligible influence. After
describing our evaluation settings (§5.1), we discuss our results for the
micro- (§5.2) and macro-benchmarks (§5.3).

###  5.1. Experimental Setup

We used a machine equipped with an Intel® Xeon® E-2288G CPU @ $3.70$ GHz ($8$
physical cores and activated hyper-threading) and $128$ GiB of RAM. The system
uses a $2$ TB Intel® 660p M.2 using PCIe 3.0$x$4 as persistent storage. The
operating system is Ubuntu 18.04.5 LTS with Linux 4.15.0-142 and SGX software
support is made available through the SGX OOT driver in the most recent
version 2.11 built from source (Intel, 2021). We compare SinClave’s
performance with the master version of SCONE v5.3.0 as baseline. SinClave’s
implementation is built on top of the same version of SCONE.

###  5.2. Micro-benchmarks

All micro-benchmarks have been executed with the Criterion benchmarking
framework (Heisler, 2021). The framework was configured to warm each benchmark
for $3$ seconds and afterward perform measurements for at least $120$ seconds
taking $20$ samples. Variance is omitted as it was negligible throughout all
benchmarks. We report the mean runtime as reported by Criterion.

SHA-256 Computation Performance. We investigate the overhead introduced by
SinClave’s interruptible SHA-256 implementation through evaluation of the
hashing performance. The benchmark hashes a buffer of various sizes and
measures the throughput. The buffer sizes correspond to differently sized SGX
enclaves, i.e., the size of an enclave program that must be hashed (measured)
for attestation.

Figure 6. Calculation of a SHA256 checksum with different implementations.

A plot showing the evaluation of different implementations calculating a
SHA256 checksum.

(a) Compilation duration.

(b) Sigstruct Signing and Verification.

(c) SinClave operation durations.

Figure 7. Micro-benchmarks of SinClave.

No description

Figure 6 shows a comparison of three SHA256 checksum calculations. The
baseline for this comparison is Ring v0.16.20 (Smith, 2021), a crypto library
for Rust that uses highly optimized code comparable to OpenSSL222A single run
of openssl speed sha256 reveals a speed of about $480$ MB/s for a $16$ KB
buffer..

SinClave-BaseHash indicates the implementation of SHA-256 that produces a base
enclave hash, i.e., the SHA-256 computation is interrupted before finalization
of an enclave, and the Base Enclave Hash is encoded instead of finalizing the
SHA computation. SinClave indicates the variant of SinClave’s SHA-256
implementation without interruptions, i.e., no base enclave hash is calculated
but the SHA-256 computation is finalized directly.

The results show that SHA-256 implementation in the Ring library outperforms
the interruptible variant of SinClave for all buffer sizes. Independent of the
buffer size it yields about $405$ MB/s. The SinClave implementation, which is
directly comparable to Ring, yields about $180$ MB/s. This overhead is the
result of SinClave using less optimized code. SinClave-BaseHash basically
shows an equivalent performance compared to SinClave for sufficiently large
buffer sizes. For small buffers, the overhead introduced by the additional
encoding is significantly smaller than the effort necessary for SHA-256
finalization and, thus, SinClave-BaseHash performs better with about 245 MB/s
for 2 KB buffers.

Lastly, we measured the time it takes to finalize an enclave base hash into an
enclave measurement. This time is not influenced by the buffer size and
requires constant $32$ $\mu{}$s.

Enclave Program Compilation. In SCONE, during compilation, the enclave’s
SigStruct is embedded into the binary’s ELF. The enclave measurement, which is
part of the SigStruct, is calculated by the SCONE signing tool using SHA-256.
For SinClave we adapted the algorithm of SHA-256 to support interruption and
to be able to measure individual enclave code based on a common base enclave
(§ 4). We evaluate the overhead introduced by our changes by measuring how
long it takes to compile a small C program that only contains a return
statement in the main function. We compare our results to compilation with the
system C compiler (native), and SCONE (baseline).

As shown in Figure 7a, baseline and SinClave introduce significant overhead.
While native compilation takes only about $0.033$ s, baseline requires $1.52$
s and SinClave $6.26$ s. SinClave needs about $4$ times longer for
compilation, while the ratio in the SHA-256 benchmark is just about $2.25$.
Apart from the SHA-256 algorithm, there are no significant changes SinClave
introduces to the signer tool but compared to the SHA-256 benchmark for which
a continuous buffer is hashed at once, i.e., with a single call, the signer
tool updates the SHA-256 computation iteratively. Thus, we attribute this
increased overhead to the less optimized code handling the entering and
exiting of the hash computation.

SigStruct Signing and Verification. In SinClave each enclave’s measurement is
unique and only known at execution time. SGX, on the other hand, requires a
SigStruct that is matching the enclave measurement. Thus, in SinClave
individual SigStruct must be created for each singleton enclave that is
started, which is done by the verifier, i.e., SCONE CAS (§4.4). SGX SigStructs
are signed with an RSA key with $3072$ bits. We assess the introduced overhead
by measuring the time it takes to sign the SigStruct. During On-Demand
SigStruct generation, the verifier must assess the authenticity of received
SigStructs to prevent tampering, therefore we also measure the time it takes
to verify a SigStruct.

As shown in Figure 7b, SigStruct signing takes $4.9$ ms on average. While
verification takes $0.4$ ms on average (“Verify C.” in Figure 7b). We further
measure the time the verification takes in case it fails “Verify E.”. The
results show no difference to correct verification.

Singleton Page Retrieval. In this experiment, we measure the entire overhead
introduced by singleton page retrieval. Singleton page retrieval is the most
distinct modification to the attestation procedure since this interaction is
introduced by SinClave. Thus, we expect it to dominate the overhead introduced
by SinClave.

We measured the entire time it takes to retrieve a singleton page from the
verifier, as well as, the time it takes to connect to the verifier and
terminate the connection again after a simple no-op request. Figure 7c shows
the results of these measurements split into individual operations necessary
to complete the retrieval. We use measurements from section §5.2 to attribute
the latency.

The entire operation adds about $26.3$ ms to each enclave’s start. During this
operation, a negligible amount of time is spent to verify the received
SigStruct’s integrity ($0.4$ ms) and to calculate the expected singleton’s
enclave measurement ($32$ $\mu{}$s). Establishing the connection (O/C)
requires $3.74$ ms and the signing of the singleton’s SigStruct $4.93$ ms. The
majority of time is spent with miscellaneous other necessary activities in the
SCONE CAS. These involve the loading and parsing of the configuration details
from the encrypted database, as well as, the enforcement of access policies.

Figure 8. Measurement of program execution.

No description.

Figure 9. The performance overhead of SinClave with real-world workloads.

No description

Enclave Execution. In our final micro-benchmark, we evaluate the overhead
SinClave introduces for running attested SGX enclaves in total. For this
benchmark, we use the minimal C program from Section 5.2 and measure its
execution time under various configurations. We ran it in simulation mode,
i.e., without hardware protection, with SGX hardware protection, and with
hardware and attestation. Also, we vary the enclave heap size from $32$ MB to
$2$ GB. We compare the individual measurements with each other to estimate the
overheads introduced by the framework, the hardware protection, and the
attestation procedure.

Figure 9 shows the results for SinClave and baseline. We find that, throughout
the benchmarks, SinClave is slower only within single digit microsecond values
compared to the baseline for execution modes that make no use of attestation.
Thus, we conclude that SinClave does not introduce significant overhead to the
framework’s nor the hardware protection’s overheads.

Attestation overhead increases slightly with enclave heap size, which is an
effect of increased cache pressure. For attested execution SinClave introduces
$132$ ms to $144.2$ ms compared to $36.3$ ms to $65.9$ ms introduced by the
baseline system. Compared to the overall execution time of up to $5.2$ s (min:
$258.1$ ms), these overheads are only significant in experiments with small
enclave heap size.

###  5.3. Macro-benchmarks

In this section, we evaluate our protection mechanism overhead using real-
world applications including _(1)_ a simple python application with an
encrypted volume (Python) (UG, 2021), _(2)_ image classification using
OpenVino (Toolkit, 2020) (OpenVino), and _(3)_ Pytorch CIFAR-10 dataset
training (PyTorch, 2021) (Pytorch). We measure the duration of these workloads
when running them inside SGX enclaves with SinClave using the SCONE
attestation service CAS. We then enable the proposed protection mechanism
SinClave in SCONE and perform the same experiments as baseline to evaluate the
impact of the protection mechanism on performance as shown in Figure 9.
SinClave introduces $1.03$%, $2.49$%, and $13.2$% overheads in the benchmarks
for Python, Openvino, Pytorch, in comparison to the baseline. Our benchmarks
show that for real-world applications the overhead measured during the micro-
benchmarks is negligible.

##  6\. Related work

Attacks against remote attestation. The remote attestation in several TEEs is
currently vulnerable to _relay attacks_. In these attacks, an adversary can
control the OS (or other software) on the target host/device to relay incoming
attestation requests to another host/device. For example, in the context of
TPM attestation, the relay attack is called _cuckoo attack_ (Parno, 2008). In
this attack, the adversary uses malicious software on the target host to
redirect the attestation to a host that he controls. Thus, the provisioning of
confidential data is directed to the host controlled by the adversary. To
handle this attack, several defense mechanisms have been proposed however they
still contain several limitations (Fink et al., 2011). For example, they
require dedicated measurement conditions, e.g., turn off of some OS processes.
In addition, it has a high amount of false positives.

Meanwhile, in another approach Flicker (McCune et al., 2008) attests to only a
single application at the time, requiring to shut down the entire OS and all
other services and also requires a trusted administrator. Recently, several
attacks against Intel SGX remote attestation have been presented: ProximiTEE
(Dhar et al., 2020) presents a relay attack on SGX attestation, or SGAxe (van
Schaik et al., 2020) shows an attack that bypasses the SGX remote
attestations. However these attacks require to first perform micro-
architectural attacks such as side-channel attacks (Kocher et al., 2019; Chen
et al., 2019; Bulck et al., 2018; Lipp et al., 2018), thereafter use the
gained secrets to attack the remote attestation. In this paper, we present an
attack against TEEs remote attestation without requiring to perform any
microarchitectural attacks.

Side-channel attacks. Researchers recently demonstrated that TEEs are
vulnerable to side-channel attacks (Kocher et al., 2019; Chen et al., 2019;
Bulck et al., 2018; Lipp et al., 2018). To mitigate these attacks, protection
mechanisms have been proposed (Brasser et al., 2019; Ren et al., 2015;
Stefanov et al., 2018; Gruss et al., 2017; Rane et al., 2015; Deng et al.,
2019). They follow the idea of hiding the memory access pattern of
applications running inside SGX enclaves (Gruss et al., 2017; Ren et al.,
2015; Stefanov et al., 2018; Brasser et al., 2019) by generating random memory
accesses or shuffling the encrypted memory elements in a way that the
observable access patterns are similar to each other and independent from the
original access patterns. Intel has been also working to release various
patches, microcode updates to mitigate SGX side-channel attacks. In this
paper, we however turn our focus to the second key feature of TEEs, which is
the _remote attestation_ mechanism. We present an attack that bypasses the
remote attestation protocol. Our attack does not require attacking the sealing
mechanism of TEEs as the previous work (van Schaik et al., 2020).

Support legacy applications. Several SGX platforms (Arnautov et al., 2016;
Priebe et al., 2019; Tsai et al., 2017; Shen et al., 2020b) have been
developed to support legacy applications running with Intel SGX without
changing their source code and are widely used in practice (Arnautov et al.,
2016). SCONE (Arnautov et al., 2016) was the first framework that supports
switchless system calls to improve the performance of applications running
with Intel SGX. It supports porting many confidential cloud-native
applications, including data analytics systems (Ozga et al., 2021; Quoc et
al., 2020; Le Quoc et al., 2019; Quoc and Fetzer, 2021), storage systems
(Bailleu et al., 2021; Krahn et al., 2018), key management systems (Gregor et
al., 2020), and performance monitoring (Krahn et al., 2020). Following this
idea, other SGX frameworks such as Graphene-SGX (Tsai et al., 2017), SGX-LKL
(Priebe et al., 2019), or Occlum (Shen et al., 2020b) have been proposed.
Unfortunately, Graphene-SGX and Occlum only provide the building blocks for
users to perform remote attestation and secret provisioning for their
applications. SCONE and SGX-LKL support a transparently performing remote
attestation mechanism for users. However, this paper demonstrates that the
supported mechanism is not secure by showing an attack to bypass the remote
attestation. We also provide novel protection against this attack and
strengthen the remote attestation mechanism of SCONE with only lightweight
overhead.

##  7\. Conclusion

In this paper, we show that the measurement for remote attestation in TEEs is
currently vulnerable to certain “resue” attacks of enclaves since it only
guarantees software integrity before it runs, but not during runtime. We
demonstrate a such practical attack against different TEE frameworks. In
addition, we provide a solution to strengthen the measurement by considering
all aspects that determine the behavior of remote applications and ensure the
measurement is fresh. We evaluate our protection mechanism with real-world
applications. The evaluation shows that it can protect against reuse attacks
and incurs only negligible overhead, which is only from $1.03$% to $13.2$%
depending on running applications.

Acknowledgements. We thank Andrey Brito, our reviewers, and our shepherd
Valerio Schiavoni for their work. This publication was funded by the European
Union’s Horizon Europe research and innovation program under grant agreements
101092646 (Cloudskin), 101092644 (Neardata), 101016577 (AI-Sprint), by the
Deutsche Forschungsgemeinschaft (DFG) in subproject B07 of the Collaborative
Research Center Transregio 96 (project 174223256), and by the Federal Ministry
of Education and Research (6G-Life).

## References

  * (1)
  * Anati et al. (2013) Ittai Anati, Shay Gueron, P. Simon Johnson, and R. Vincent Scarlata. 2013.  Innovative technology for CPU based attestation and sealing. In _Proceedings of the 2nd International Workshop on Hardware and Architectural Support for Security and Privacy (HASP)_.
  * Arnautov et al. (2016) Sergei Arnautov, Bohdan Trach, Franz Gregor, Thomas Knauth, Andre Martin, Christian Priebe, Joshua Lind, Divya Muthukumaran, Dan O’Keeffe, Mark L Stillwell, et al. 2016\.  SCONE: Secure Linux Containers with Intel SGX. In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_.
  * Bailleu et al. (2021) Maurice Bailleu, Dimitra Giantsidi, Vasilis Gavrielatos, Vijay Nagarajan, Pramod Bhatotia, et al. 2021\.  Avocado: A Secure $\\{$In-Memory$\\}$ Distributed Storage System. In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_.
  * Baumann et al. (2015) Andrew Baumann, Marcus Peinado, and Galen Hunt. 2015.  Shielding applications from an untrusted cloud with haven.  _ACM Transactions on Computer Systems (TOCS)_ (2015).
  * Brasser et al. (2019) Ferdinand Brasser, Srdjan Capkun, Alexandra Dmitrienko, Tommaso Frassetto, Kari Kostiainen, and Ahmad-Reza Sadeghi. 2019.  DR.SGX: Automated and Adjustable Side-Channel Protection for SGX Using Data Location Randomization. In _Proceedings of the 35th Annual Computer Security Applications Conference (ACSAC)_.
  * Bulck et al. (2018) Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and Raoul Strackx. 2018.  Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution. In _27th USENIX Security Symposium (USENIX Security 18)_.
  * Chen et al. (2019) G. Chen, S. Chen, Y. Xiao, Y. Zhang, Z. Lin, and T. H. Lai. 2019.  SgxPectre: Stealing Intel Secrets from SGX Enclaves Via Speculative Execution. In _IEEE European Symposium on Security and Privacy (Euro S &P)_.
  * Corp ([n. d.]) Intel Corp. [n. d.].  Gramine: A Linux-compatible Library OS for Multi-Process Applications.  hhttps://github.com/gramineproject/gramine.  Accessed: May 2023.
  * Costan and Devadas (2016a) Victor Costan and Srinivas Devadas. 2016a.  Intel SGX Explained.  _IACR Cryptol. ePrint Arch._ (2016).
  * Costan and Devadas (2016b) Victor Costan and Srinivas Devadas. 2016b.  Intel SGX Explained.  _IACR Cryptology ePrint Archive_ (2016).
  * Deng et al. (2019) Shuwen Deng, Wenjie Xiong, and Jakub Szefer. 2019.  Analysis of Secure Caches using a Three-Step Model for Timing-Based Attacks.  Journal of Hardware and Systems Security volume 3.
  * Dhar et al. (2020) Aritra Dhar, Ivan Puddu, Kari Kostiainen, and Srdjan Capkun. 2020.  ProximiTEE: Hardened SGX attestation by proximity verification. In _Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy_.
  * Fink et al. (2011) Russell A Fink, Alan T Sherman, Alexander O Mitchell, and David C Challener. 2011.  Catching the cuckoo: Verifying tpm proximity using a quote timing side-channel. In _International Conference on Trust and Trustworthy Computing_.
  * Gordon (2018) James C Gordon. 2018.  Microsoft Azure Confidential Computing with Intel SGX.  https://software.intel.com/content/www/us/en/develop/blogs/microsoft-azure-confidential-computing-with-intel-sgx.html.  Accessed: May 2023.
  * Gregor et al. (2020) Franz Gregor, Wojciech Ozga, Sébastien Vaucher, Rafael Pires, Sergei Arnautov, André Martin, Valerio Schiavoni, Pascal Felber, Christof Fetzer, et al. 2020\.  Trust management as a service: Enabling trusted execution in the face of byzantine stakeholders. In _2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)_.
  * Gruss et al. (2017) Daniel Gruss, Julian Lettner, Felix Schuster, Olya Ohrimenko, Istvan Haller, and Manuel Costa. 2017.  Strong and efficient cache side-channel protection using hardware transactional memory. In _Proceedings of the 26th USENIX Security Symposium (USENIX Security 17)_.
  * Gulley et al. (2013) Sean Gulley, Vinodh Gopal, Kirk Yap, Wajdi Feghali, Jim Guilford, and Gil Wolrich. 2013.  Intel sha extensions–new instructions supporting the secure hash algorithm on intel architecture processor.  _Intel White Paper_ (2013).
  * Heisler (2021) Brook Heisler. 2021.  Criterion.rs - Statistics-driven Microbenchmarking in Rust.  https://github.com/bheisler/criterion.rs.  Accessed: May 2023.
  * Intel (2020) Intel. 2020.  Intel® 64 and ia-32 architectures software developer’s manual.  https://software.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-sdm-volume-3d-system-programming-guide-part-4.html.  _Volume 3D: System programming Guide, Part 4_ (2020).  Accessed: May 2023.
  * Intel (2021) Intel. 2021.  Intel SGX Linux(R) Driver.  https://github.com/intel/linux-sgx-driver.  Accessed: May 2023.
  * Intel Corporation (2020) Intel Corporation. 2020.  _An introductory overview of the Intel TDX technology_.  Intel White Paper. Intel Corporation.
  * Karnati (2018) Pratheek Karnati. 2018.  Data-in-use protection on IBM Cloud using Intel SGX.  https://www.ibm.com/cloud/blog/data-use-protection-ibm-cloud-using-intel-sgx.  Accessed: May 2023.
  * Kocher et al. (2019) Paul Kocher, Jann Horn, Anders Fogh, , Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, and Yuval Yarom. 2019.  Spectre Attacks: Exploiting Speculative Execution. In _40th IEEE Symposium on Security and Privacy (S &P’19)_.
  * Krahn et al. (2020) Robert Krahn, Donald Dragoti, Franz Gregor, Do Le Quoc, Valerio Schiavoni, Pascal Felber, Clenimar Souza, Andrey Brito, and Christof Fetzer. 2020.  TEEMon: A continuous performance monitoring framework for TEEs. In _Proceedings of the 21st International Middleware Conference_.
  * Krahn et al. (2018) Robert Krahn, Bohdan Trach, Anjo Vahldiek-Oberwagner, Thomas Knauth, Pramod Bhatotia, and Christof Fetzer. 2018.  Pesos: Policy enhanced secure object store. In _Proceedings of the Thirteenth EuroSys Conference_.
  * Le Quoc et al. (2019) Do Le Quoc, Franz Gregor, Jatinder Singh, and Christof Fetzer. 2019.  Sgx-pyspark: Secure distributed data analytics. In _The World Wide Web Conference (WWW)_.
  * Lipp et al. (2018) Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Anders Fogh, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, Yuval Yarom, and Mike Hamburg. 2018.  Meltdown: Reading Kernel Memory from User Space. In _27th USENIX Security Symposium (USENIX Security 18)_.
  * McCune et al. (2008) Jonathan M. McCune, Bryan J. Parno, Adrian Perrig, Michael K. Reiter, and Hiroshi Isozaki. 2008.  Flicker: An Execution Infrastructure for Tcb Minimization. In _Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems (EuroSys)_.
  * McKeen et al. (2013) Frank McKeen, Ilya Alexandrovich, Alex Berenzon, Carlos V. Rozas, Hisham Shafi, Vedvyas Shanbhogue, and Uday R. Savagaonkar. 2013.  Innovative instructions and software model for isolated execution. In _Proceedings of the 2nd International Workshop on Hardware and Architectural Support for Security and Privacy (HASP)_.
  * Merkle (1979) Ralph Charles Merkle. 1979.  _Secrecy, authentication, and public key systems._ Stanford university.
  * Ozga et al. (2021) Wojciech Ozga, Do Le Quoc, and Christof Fetzer. 2021.  Perun: Confidential Multi-stakeholder Machine Learning Framework with Hardware Acceleration Support. In _IFIP Annual Conference on Data and Applications Security and Privacy_.
  * Parno (2008) Bryan Parno. 2008.  Bootstrapping Trust in a ”Trusted” Platform. In _Proceedings of the 3rd Conference on Hot Topics in Security_.
  * Penard and van Werkhoven (2008) Wouter Penard and Tim van Werkhoven. 2008.  On the secure hash algorithm family.  _Cryptography in context_ (2008).
  * Priebe et al. (2019) Christian Priebe, Divya Muthukumaran, Joshua Lind, Huanzhou Zhu, Shujie Cui, Vasily A Sartakov, and Peter Pietzuch. 2019.  SGX-LKL: Securing the Host OS Interface for Trusted Execution.  _arXiv preprint arXiv:1908.11143_ (2019).
  * PUB (2012) FIPS PUB. 2012.  Secure hash standard (shs).  _Fips pub_ (2012).
  * PyTorch (2021) PyTorch. 2021.  Training a Classifier.  https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html.  Accessed: May 2023.
  * Quoc and Fetzer (2021) Do Le Quoc and Christof Fetzer. 2021.  Secfl: Confidential federated learning using tees.  _arXiv preprint arXiv:2110.00981_ (2021).
  * Quoc et al. (2020) Do Le Quoc, Franz Gregor, Sergei Arnautov, Roland Kunkel, Pramod Bhatotia, and Christof Fetzer. 2020.  Securetf: A secure tensorflow framework. In _Proceedings of the 21st International Middleware Conference_.
  * Rane et al. (2015) Ashay Rane, Calvin Lin, and Mohit Tiwari. 2015.  Raccoon: Closing digital side-channels through obfuscated execution. In _Proceedings of the 24th USENIX Security Symposium (USENIX Security 15)_.
  * Ren et al. (2015) Ling Ren, Christopher Fletcher, Albert Kwon, Emil Stefanov, Elaine Shi, Marten Van Dijk, and Srinivas Devadas. 2015.  Constants Count: Practical Improvements to Oblivious RAM. In _Proceedings of the 24th USENIX Security Symposium (USENIX Security 15)_.
  * Sev-Snp (2020) AMD Sev-Snp. 2020.  Strengthening VM isolation with integrity protection and more.  _White Paper, January_ (2020), 8.
  * Shen et al. (2020a) Youren Shen, Hongliang Tian, Yu Chen, Kang Chen, Runji Wang, Yi Xu, Yubin Xia, and Shoumeng Yan. 2020a.  Occlum: Secure and efficient multitasking inside a single enclave of intel sgx. In _Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems_.
  * Shen et al. (2020b) Youren Shen, Hongliang Tian, Yu Chen, Kang Chen, Runji Wang, Yi Xu, Yubin Xia, and Shoumeng Yan. 2020b.  Occlum: Secure and efficient multitasking inside a single enclave of intel sgx. In _Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_.
  * Singh et al. (2020) Jatinder Singh, Jennifer Cobbe, Do Le Quoc, and Zahra Tarkhani. 2020.  Enclaves in the clouds: legal considerations and broader implications.  _Queue_ (2020), 78–114.
  * Singh et al. (2021) Jatinder Singh, Jennifer Cobbe, Do Le Quoc, and Zahra Tarkhani. 2021.  Enclaves in the clouds: Legal considerations and broader implications.  _Commun. ACM_ (2021), 42–51.
  * Smith (2021) Brian Smith. 2021.  Safe, fast, small crypto using Rust.  https://github.com/briansmith/ring.  Accessed: May 2023.
  * Stefanov et al. (2018) Emil Stefanov, Marten Van Dijk, Elaine Shi, T-H Hubert Chan, Christopher Fletcher, Ling Ren, Xiangyao Yu, and Srinivas Devadas. 2018.  Path oram: An extremely simple oblivious ram protocol. In _Journal of the ACM (JACM)_.
  * Toolkit (2020) OpenVino Toolkit. 2020.  Security Barrier Camera Demo.  https://docs.openvinotoolkit.org/2020.1/_demos_security_barrier_camera_demo_README.html.  Accessed: May 2023.
  * Tsai et al. (2017) Chia-Che Tsai, Donald E Porter, and Mona Vij. 2017.  Graphene-SGX: A Practical Library OS for Unmodified Applications on SGX. In _USENIX Annual Technical Conference (USENIXATC 17)_.
  * UG (2021) Scontain UG. 2021.  Python with Encrypted Volume Demo.  https://github.com/scontain/volume-demo.  Accessed: May 2023.
  * van Schaik et al. (2020) Stephan van Schaik, Andrew Kwong, Daniel Genkin, and Yuval Yarom. 2020.  SGAxe: How SGX fails in practice.
  * Winter (2008) Johannes Winter. 2008.  Trusted computing building blocks for embedded linux-based ARM trustzone platforms. In _Proceedings of the 3rd ACM workshop on Scalable trusted computing (STC)_.