# Trust Management as a Service:
Enabling Trusted Execution in the Face of Byzantine Stakeholders

Franz Gregor1, Wojciech Ozga1, Sébastien Vaucher2, Rafael Pires2, Do Le Quoc1,
Sergei Arnautov3,
André Martin1, Valerio Schiavoni2, Pascal Felber2, Christof Fetzer1  TU
Dresden, Germany1 — Université de Neuchâtel, Switzerland2 — Scontain UG,
Germany3

###### Abstract

Trust is arguably the most important challenge for critical services both
deployed as well as accessed remotely over the network. These systems are
exposed to a wide diversity of threats, ranging from bugs to exploits, active
attacks, rogue operators, or simply careless administrators. To protect such
applications, one needs to guarantee that they are properly configured and
securely provisioned with the “secrets” (, encryption keys) necessary to
preserve not only the confidentiality, integrity and freshness of their data
but also their code. Furthermore, these secrets should not be kept under the
control of a single stakeholder—which might be compromised and would represent
a single point of failure—and they must be protected across software versions
in the sense that attackers cannot get access to them via malicious updates.
Traditional approaches for solving these challenges often use _ad hoc_
techniques and ultimately rely on a hardware security module (HSM) as root of
trust. We propose a more powerful and generic approach to trust management
that instead relies on trusted execution environments (TEEs) and a set of
stakeholders as root of trust. Our system, Palæmon, can operate as a managed
service deployed in an untrusted environment, , one can delegate its
operations to an untrusted cloud provider with the guarantee that data will
remain confidential despite not trusting any individual human (even with root
access) nor system software. Palæmon addresses in a secure, efficient and
cost-effective way five main challenges faced when developing trusted
networked applications and services. Our evaluation on a range of benchmarks
and real applications shows that Palæmon performs efficiently and can protect
secrets of services without any change to their source code.

##  I Introduction

Protecting the confidentiality, integrity and freshness (CIF) of application
data is a key challenge of many applications, and a primary reason for
companies to be wary of deploying their system outside premises in shared
environments. To illustrate the challenges faced in such scenarios, consider
for instance modern machine learning applications that require significant
computing power and would hence benefit from running in a scalable cloud
infrastructure. Yet, at the same time, they need to protect their code, their
training data (input) and the produced model (output), all of which represent
key assets for their respective owners (see Fig. 1). The development and
operation of such a large system involve multiple stakeholders, notably
software developers, system administrators, data providers and cloud
providers, which cannot necessarily be trusted and might collude to gain
advantages over the other stakeholders [54]. For example, we cannot trust that
system administrators or software developers will neither leak [63, 24] nor
modify application code and data. To address these challenges and enable
_trusted application execution in the face of Byzantine stakeholders_ , we
have designed Palæmon, a trust management service that builds on top of the
SCONE platform [5] and ultimately relies on hardware-based trusted execution
environments (TEEs) for secure execution.

Fig. 1: Components and stakeholders of the ML use case. The training data and
model belong to the data and model providers, whereas the ML processing
runtime is owned by the software provider. These key assets must be protected
at all times from the other stakeholders.

Palæmon was originally developed in order to address real problems from some
users and there was no alternative service available to solve the problems we
faced. It was extended and refined over the course of the last two years to
take into account additional threats and support the evolution of trusted
applications via secure updates. We motivate its design and illustrate its
operation on a real-life production use case in the machine learning space,
yet our approach is more general and applies to a much wider range of
applications.

At the core of Palæmon is a _trust management service_ specifically designed
to address the following main challenges:

  1. 1.

Secret management — How can we securely provide applications with secrets in
an untrusted environment?

  2. 2.

Managed operation — How can we delegate the management of Palæmon to untrusted
stakeholders?

  3. 3.

Robust root of trust — How can we protect CIF against malicious stakeholders?

  4. 4.

Rollback protection — How can we ensure freshness of data and code in an
efficient manner?

  5. 5.

Secure update — How can we support secure updates of applications and Palæmon?

To better understand the importance of these challenges, let us illustrate
them on one of the original use cases that guided the design of Palæmon. This
real-life use case comes from a company, a _software provider_ specialized in
ML that develops its ML engine in Python (see Fig. 1). The engine is executed
by a second party, the _model provider_ , which processes training data to
produce a model. The software provider must neither learn the training data
nor the produced model. Conversely, the model provider must not learn the
application code of the software provider. Moreover, the software provider may
want to limit the number of models produced by its application, and hence the
number of times the code is executed. The model provider might try to
circumvent this limitation by reverting the application to a previous state in
order to generate more models (“rollback attack”).

In its initial deployment, the model provider ran the ML engine on dedicated
computers within its own air-gapped infrastructure, over which it has complete
control. Then, the model provider wants to execute the application in a cloud
instead. Therefore, it also needs to protect its training data and generated
models from the third-party cloud provider, including developers and system
administrators. Finally, when operating in the untrusted cloud, it should be
possible for the software provider as well as the provider of the Python
runtime to continuously update their software while preserving the trust
guarantees, , one should prevent an attacker from injecting malicious code
during software updates.

This real-life application illustrates the need for addressing the
aforementioned set of problems: it requires secure _secret management_ to
preserve confidentiality of the code and data, _managed operation_ to delegate
the management to a cloud provider, a _robust root of trust_ to protect both
data and code, _rollback protection_ to control the number of models produced,
and support for _secure update_ for managing the life cycle of software
deployed in a third-party cloud. Our general approach to address these five
problems is to define a novel trust management service (TMS) that supports
security policies and is able to deal with untrusted stakeholders. The TMS can
itself be managed by an untrusted entity (, the ML model provider) while still
being trusted by other entities (, the ML software provider).

The contributions of this paper are as follows. While many of the techniques
that we use are known, combining these to address the problems we face is
novel. We are not aware of any other service that transparently protects
applications from rollback attacks with little overheads, supports secure
software updates and guarantees CIF of data and code even in the face of
insider attacks, while still being able to delegate the management of the
service to remote providers. Throughput of our monotonic counters is 5 orders
of magnitude higher than those provided by the SGX platform.

The remainder of the paper is organized as follows. We first introduce the
threat model and problem in §II. We then present the architecture of Palæmon
in §III and its implementation in §IV. We evaluate Palæmon’s security and
performance in §V, present a real use case in production settings in §VI,
discuss related work in §VII and conclude in §VIII.

##  II Threats and Challenges

We first introduce our threat model before we describe in greater details the
challenges addressed by Palæmon.

###  II-A Threat Model

Services executing in untrusted environments such as clouds are vulnerable to
attackers with root privileges. Attackers often target the credentials of
system administrators to gain access to hosts [75, 26, 22]. They also exploit
bugs in the system software to gain root privileges on systems [28, 19].

Multiple stakeholders of the same services cannot trust each other to protect
the CIF of their digital assets such as data and code. Fig. 2 illustrates the
security goals of Palæmon with regard to the stakeholders as well as the
building blocks in our system. For example, Palæmon can protect CIF of Python
code deployed at remote sites, a somewhat surprising but popular requirement
for several applications of our users. Furthermore, applications must be
regularly updated, and we need to protect them against malicious software
updates triggered by attackers.

Note that we cannot trust any system administrator or software
developer—actually not any single individual. Hence, we do not trust in CIF of
main memory. We also do not assume that the OS-based access control can ensure
CIF since a single malicious system administrator could break this assumption.
It follows that one cannot trust software updates originating from a single
developer nor security policies defined by any individual, independent of
their authorization level or trustworthiness.

Palæmon protects from attempts to compromise CIF of code and data by requiring
a quorum of trusted entities—not just a single individual—to approve any given
change. This is based on the assumption that, in any organisation that must
securely operate an application, one can identify a set of $n$ stakeholders
and a threshold $f$ (with $f\\!{<}n$), such that $n{-}\\!f$ stakeholders can
be trusted at any point in time, , at most $f$ of them exhibit Byzantine
behaviour because of neglect or malicious intent. Hence, if at least
$f\\!{+}1$ stakeholders approve a change, such as a software update, at least
one of them judged it to be trustworthy. In practice, the typical convention
is that any policy change must be approved by all members, , stakeholders in a
legal business contract. A single member can choose to decline and in this way
prevent malicious policy changes.

TEEs such as Intel software guard extensions (SGX) are typically vulnerable to
side-channel attacks [9, 46, 18, 70, 72]. Such attacks can be addressed using
existing techniques (, _Varys_ [57]) and are out of the scope of this work.
Similarly, we do not consider denial of service attacks.

In our threat model, we anticipate that new attacks on TEEs can appear in the
future. We assume that we can put mitigation measures either in software or in
microcode, or by limiting execution to certain CPU types and features not
vulnerable to these attacks. This implies that we need to be able to
continuously update Palæmon as well as the applications and deactivate
vulnerable instances within a short period of time, so that the system is
protected against new attacks.

###  II-B Problem Statement

We now introduce a more detailed definition of the problem, along with five
challenges identified when analysing and operating a wide range of real-world
applications that we briefly introduced earlier, with regard to our ML use
case.

Secret management — Legacy software can use program arguments, environment
variables or files to obtain secrets, as can be observed in Table I, which
shows a quick analysis of various popular services. To account for this
diversity and provide seamless integration of secret management in legacy
applications, we need to solve the problem of:

How to support secret management for common configuration approaches in a
secure way and without requiring modifications to the source code?

Container images (e.g., Docker images) are a popular way to deploy
applications. We want to be able to customize container images such that not
only  _(i)_ different application developers can inject different secrets in
their derived application images, but also  _(ii)_ one can inject different
secrets in each container instance of an image (see Fig. 2).  For example, a
client running such an image might inject client-specific secrets for the
application to be able to decrypt client-encrypted input files. The injection
mechanism must protect CIF in the sense that an adversary cannot read, modify
or replace these secrets.

Table I: How popular services obtain secrets (∗: evaluated in §V). Program |  Version |  Lang. |  Args. |  Env. |  Files
---|---|---|---|---|---
Consul |  1.2.3 |  Go |  ✗ |  ✓ |  ✓
MariaDB∗ |  10.1.26 |  C/C++ |  ✓ |  ✓ |  ✓
Memcached∗ |  1.5.6 |  C |  ✗ |  ✗ |  ✗
MongoDB |  4.0 |  C++ |  ✓ |  ✓ |  ✓
Nginx∗ |  2.4 |  C |  ✓ |  ✓ |  ✓
PostgreSQL |  10.5 |  C |  ✓ |  ✓ |  ✓
Redis |  4.0.11 |  C |  ✗ |  ✗ |  ✓
Vault∗ |  0.8.1 |  Go |  ✓ |  ✗ |  ✓
WordPress |  4.9.x |  PHP |  ✗ |  ✗ |  ✓
ZooKeeper∗ |  3.4.11 |  Java |  ✗ |  ✗ |  ✓

Managed operation — The behaviour of applications is not only determined by
the application code, but also by its configuration parameters such as
configuration files. Some applications could be configured in a way that can
leak confidential data. Therefore, clients would need the ability to verify
that an application is properly configured to ensure CIF of their data. This
challenge is especially critical for Palæmon, as it manages secrets on behalf
of clients while operating in a different administrative domain. Therefore, in
this paper, we address the problem of:

How to delegate the management of applications, as well as Palæmon instances,
to untrusted providers while still ensuring CIF of the secrets?

Robust root of trust — In addition to cloud providers and system
administrators, we do not trust insiders such as the software developers who
build the software components, or the security experts who design the security
policies. We must therefore solve the problem of:

How to guarantee CIF of data and code even in face of malicious insiders, , in
a Byzantine environment?

Rollback protection — Whereas cryptography can preserve confidentiality and
integrity of data via encryption, it does not protect from the powerful class
of “rollback” attacks by which a malicious party attempts to replace the
current state of the file system with a previous version. In this way, they
can revert data and undo some processing. In our ML use case, a client could
roll back the file system to execute the application more often than
permitted. Preventing rollbacks typically implies significant runtime
overheads and application reengineering, which we want to avoid. The problem
we address is hence:

How to protect applications from rollback attacks with only negligible
overhead and without requiring modifications to the source code?

Secure update — Software needs to be updated continuously, not only for adding
new features but more importantly to fix bugs and patch security
vulnerabilities. We therefore need a secure approach to update applications,
by making sure that the new versions are genuine before transferring the
secrets of the previous version to the new one. Specifically:

How to update applications, as well as Palæmon itself, without compromising
secrets even when facing a malicious software update initiated by an insider?

Moreover, this should be supported in settings where the management of Palæmon
is delegated to an untrusted party that is permitted to perform the update.

Fig. 2: Stakeholders, components and protection objectives of Palæmon. White
on black text refers to the motivating example of the machine learning
application.

##  III Approach: A Trust Management Service

Here we describe how Palæmon tackles the introduced problems above, with more
technical descriptions also given in §IV.

Fig. 3: A policy defines which applications can access which secrets on which
hosts.

Fig. 4: Managed Palæmon supports certificate-based attestation via Palæmon CA.

Fig. 5: Application rollbacks are detected by maintaining _expected tags_ at
Palæmon.

###  III-A Secret Management

One of the main roles of Palæmon is to pass secrets in a trusted manner to
applications after attesting them. Each application is executed in a TEE and
associated with a _security policy_ that defines _which applications can
access which secrets on which hosts_. Applications are identified by a
cryptographic hash of the enclave [20] (MRE) and the content of the files they
can access. Secrets are typed and can either be explicitly defined, or
randomly chosen by Palæmon.

⬇

List 1: A Palaemon Policy Example

{

name: python_policy

services:

- name: python_app

image_name: python_image

command: python /app.py -o /encrypted-output

mrenclaves: ["$PYTHON_MRENCLAVE"]

platforms: ["$PLATFORM_ID"]

pwd: /

fspf_path: /fspf.pb

fspf_key: "$PALAEMON_FSPF_KEY"

fspf_tag: "$PALAEMON_FSPF_TAG"

images:

- name: python_image

volumes:

- name: encrypted_output_volume

path: /encrypted-output

volumes:

# an encrypted volume will

# be automatically generated

- name: encrypted_output_volume

# export encrypted volume to output policy

export: output_policy

}

Access to a security policy is guarded by a two-stage access control mechanism
using a _certificate_ and a _policy board_ (see Fig. 5). One can define the
access control and security policy in such a way that only applications under
the control of the security policy can gain access to the secrets. In this
way, one can prevent any stakeholder from accessing the secrets.

Secrets can be passed to applications as command line arguments, environment
variables, or can be injected into files. The files can contain Palæmon
variables referring to the names of secrets defined in the security policy.
The variables are transparently replaced by the value of the secret when an
application that is permitted to access the secrets reads the file. By
_transparently_ , we mean that the application is not aware of the replacement
and its code does not need to be modified.

Secret management is supported through _security policies_ , whose general
structure is shown in Fig. 5. An example of a Palæmon policy defined for a
python application is presented in List 1. Each policy has a unique name and
can define:  _(a)_ the permitted MRE (line 8) of an application (several MREs
can be specified to facilitate software updates);  _(b)_ the set of permitted
platforms (line 9) on which the application is permitted to run, or none if
permitted to run on any platform;  _(c)_ the _key_ (line 12) and _tag_ (line
13) of the file system (the _tag_ is a secure hash across all files, which are
transparently en/decrypted with the _key_ inside the TEE);  _(d)_ the command
line arguments (line 7);  _(e)_ the environment variables;  _(f)_ a set of
files to inject secrets into; and  _(g)_ imports/exports of secrets from/to
other policies (line 24).

###  III-B Managed PALÆMON

Our objective is to support a feature that we can delegate the management of a
Palæmon instance to an untrusted party, say a cloud provider, while the
clients of Palæmon can still trust that their secrets are safe and well
protected. Note that the cloud provider has full control over what code it
executes and might try to run variants of Palæmon that are wrongly configured
or have modified code. We ensure that clients connecting to a Palæmon instance
can attest it, , they can verify that this instance runs the expected
unmodified Palæmon code. Moreover, this code does not support any
configuration options that negatively influence the CIF of client data stored
in the instance.

We support two ways to attest a Palæmon instance (see Fig. 5):  _(i)_ using
transport layer security (TLS) [27, 74]; and  _(ii)_ with explicit
attestation.  The TLS-based attestation requires a trusted certification
authority (CA) with a known root certificate (RC). The CA first attests the
Palæmon instance using approach _(ii)_ to ensure that this instance runs
inside a TEE and has a correct MRE. Only then will the CA provide the instance
with a certificate signed with the RC. The CA itself runs inside of a TEE and
can be attested using explicit attestation. Entities that trust the CA can
attest the instance by checking that its TLS certificate is signed by the RC.

To support software updates of Palæmon itself, the CA includes a set of
correct MREs. The CA only signs certificates for these MREs and also limits
the duration of the certificates to ensure timely upgrades to new versions of
Palæmon. The set of MREs is stored inside of the CA’s binary, , an adversary
cannot modify the set without invalidating the MRE of the CA. Hence, deploying
a new version of Palæmon requires first to deploy a new version of the CA.
Updates of the CA itself are controlled by a Palæmon policy board consisting
of a set of stakeholders and follow the procedure described in §III-E.

Clients might not trust the CA if they do not use the current set of valid
MREs, , they only trust code instances that have been deployed some time ago,
or are not represented in the Palæmon policy board. These clients need to
attest the Palæmon instance in the same way that the CA attests instances, as
described in §IV-B. In practice, any updates of PALÆMON must be approved by
all stakeholders.

###  III-C Robust Root of Trust

Our threat model permits Byzantine behaviour of stakeholders like software
developers and system administrators. Any change to an application or its
configuration can impact the CIF of both data and code. Palæmon therefore
includes a mechanism to ensure that any security policy modification must be
approved by at least $f\\!{+}1$ stakeholders. To that end, a securities policy
can define a _policy board_ and a threshold—typically set to $f\\!{+}1$—of
policy board members that must give approval for Palæmon to permit any create,
read, update and delete (CRUD) access to the policy. Upon creation, the board
of the new policy must also approve the operation. In that way, any client can
create policies as long as they have unique names, and the policy board agrees
to take control over them.

Each policy board member is represented in the security policy by a
_certificate_ and a URL of an _approval service_ , responsible to _approve_ or
_reject_ accesses to the policy. Upon a client access, Palæmon contacts the
board members, verifies their certificates and asks them for approval of the
request via a TLS-secured REST call to their approval service.

Approval services typically run inside TEEs. In case the associated board
member is a person, they should perform a two-factor authentication with one
being based on biometric identifiers. Approval services may also consist of
services that check certain aspects of a policy, , through source code
analysis and verification of the MRE. In particular, a policy board member
could be an organisation that validates software, , perform checks on behalf
of their clients to ensure that the software associated with a certain MRE can
be trusted to protect the CIF of data.

Some policy board members can be given _veto_ rights, , they can unilaterally
reject a policy change. For example, a data provider might only provide data
to applications for which it is a policy member with veto rights. In that way,
the data provider can ensure that policy changes will not result in data
leakage.

###  III-D Rollback Protection

We want to protect applications from rollback attacks without requiring their
code to be changed. To that end, Palæmon performs transparent encryption of
files inside of the TEE. It uses a Merkle tree to verify integrity of the
files and stores its root hash in the so-called _tag_ of the file system.111In
reality, Palæmon can associate an application with multiple tags to simplify
the mapping of encrypted volumes into containers. Any change of a file will
result in a new tag, hence attempts to modify the content of the file system
or to roll back to an older version can be detected by comparing the _expected
tag_ with the _actual tag_ of the file system. Palæmon ensures that there is
no violation of integrity or freshness by verifying the value of tag on each
file system access.

In order to prevent rollback attacks, it is critical to keep the expected tag
value up-to-date. This value is kept inside of a TEE but it is lost upon crash
or when the application terminates (see Fig. 5). Our approach is to persist
the expected tag: each time  _(i)_ a file is closed;  _(ii)_ the file system
is synchronised;  _(iii)_ or the application exits,  the runtime system pushes
the _expected tag_ to Palæmon via the TLS connection that was established
during application startup to perform attestation. Palæmon stores the expected
values in its database. As these values are essential for protecting the
integrity and freshness of files, the expected tags must themselves be
protected against rollback attacks. We show in §IV how Palæmon efficiently
protects its database against rollbacks.

A policy can also define a _strict mode_ for an application. In that case,
Palæmon only permits a restart of the application if the expected tag was
properly sent upon _exit_ during the last execution of the application.
Otherwise, the restart requires an explicit update of the policy, which is
needed to adjust the tag and must in turn be approved by the policy board.

###  III-E Secure Update

Palæmon protects the CIF of both code and data. The binary code that is
initially loaded in the TEE is just _integrity-_ and _freshness-_ protected
since we can only get secrets after the initial attestation of the code. In
contrast, all code that is loaded after the start of the TEE is CIF-protected.
For example, an application can load dynamic libraries in main memory, with
these being transparently decrypted and CIF-protected by Palæmon. Code that is
loaded by interpreters and just-in-time engines is CIF-protected in the same
way.

Along the same lines, Palæmon can also perform secure software updates of an
application with the help of a policy update. Applications are typically
packaged in a container image and data (, a database) is mapped into the
container via a volume. A new version of the code results in a new MRE and
_tag_ of the container file system. The new MRE and tag must be updated within
the security policy to permit the new version to start, and this update must
be approved by the policy board.

Consider the example of an image provider who maintains an image that is
regularly updated, for example a Python interpreter running inside of a TEE.
As software is updated, old versions of the image should be disabled and new
versions enabled. To reduce the effort for applications that build upon this
image, the provider will create a security policy defining in our example the
MRE for the Python interpreter and a tag covering all the dynamic and Python
libraries. This information is exported, and can then be imported by other
security policies. Any application that uses the original image can use the
exported information in its own security policy. Additionally, the
application’s policy can limit the permitted combination of MREs and tags, ,
only allow combinations that were checked by an external service. The
application will only run with combinations that are permitted by both the
image’s and the application’s policies. The advantage of computing this
intersection is that, if the image provider removes a combination that has
become unsafe, , after discovering a vulnerability, the combination will be
automatically disallowed by the application’s policy as well.

##  IV Implementation

In this section, we describe how to address some of the challenges we faced
when implementing our approach. Our implementation is based on the SCONE
platform [5] running on top of Intel SGX [20]. However, note that PALÆMON is
designed in a generic way that can be used not only for SCONE but also for
other SGX platforms such as Graphene. We selected SCONE since it is easy to
use compared to other platforms. To run an application with Intel SGX, we just
need to compile its source code with the SCONE compiler, or just link the
binary of the application with the SCONE libc.

We also considered ARM’s TrustZone TEE [4], but it only supports a single
secure zone rather than multiple enclaves and it lacks an attestation
protocol. Meanwhile, the current version of AMD’s TEE, SME/SEV, lacks
integrity protection and is vulnerable to server-side rollback attacks [52,
34, 23]. Palæmon runs inside a TEE, , inside an SGX enclave. It is implemented
in Rust [50] to ensure strong type safety. We use an encrypted embedded SQLite
[1] database running inside the same enclave as Palæmon. We describe below how
this database is protected against rollbacks without introducing any major
performance bottlenecks.

###  IV-A Application Attestation and Configuration

Upon startup, an application is transparently linked with the SCONE runtime
and loaded inside a TEE. The runtime first attests the application with the
help of Palæmon before passing control to the application. To do so, it
creates a random key pair and gets a _report_ from a local quoting enclave
[40] that associates the public key with its MRE. The runtime sends the report
via a newly-established TLS connection to Palæmon and passes along the name of
its security policy which is stored in an unprotected environment variable.
The Palæmon instance verifies that:  _(i)_ the public key of the TLS client
certificate matches the public key of the report;  _(ii)_ the security policy
name exists and the MRE is valid for the application; and  _(iii)_ the
application runs on a permitted platform—which we can verify with the report.
If this attestation succeeds, Palæmon sends the following data to the
application: the command line arguments; the environment variables; the keys
and tags for the file system; and the set of files in which secrets should be
injected together with the secrets as key/value pairs.

The Palæmon runtime supports transparent injection of secrets into existing
configuration files via a simple variable replacement mechanism. This allows
us to inject different secrets into different instances of the same
application image, without the need to change the source code. Like all files,
they can be CIF-protected via transparent encryption by the Palæmon runtime.
The runtime injects the secrets it received from the Palæmon instance in each
file as follows. The file is first read in TEE memory, then parsed, and all
variables found are replaced by their values. Whenever the file is accessed,
it is served from memory. While sizeable files can also be stored encrypted in
main memory or on the file system, configuration files are typically small, so
we keep them in TEE memory as long as they fit.

###  IV-B PALÆMON Attestation

A client of a managed Palæmon instance must be able to ensure that the code of
Palæmon was not modified and indeed runs inside of a TEE. As a matter of fact,
we must guarantee that an infrastructure provider cannot configure Palæmon in
any way that breaks the trust given by the client in Palæmon. We enforce this
by designing Palæmon for its behaviour to depend solely on MRE, , Palæmon has
zero configuration parameters that affect its behaviour with regard to
ensuring the CIF of the data stored in the instance by the clients.

A client connecting to a Palæmon instance has to attest the instance before
performing any action, such as creating a new security policy. During the
initial startup, a Palæmon instance creates a unique public/private key pair,
as well as a random key to encrypt its file system, and stores these keys in
sealed storage [35]. During a restart (after an exit or a failure), the
instance reads the keys from sealed storage to be able to authenticate itself.
We actually use SGX to enforce that only Palæmon instances on the same
platform can read the sealed file. We show in §IV-C that at most one Palæmon
instance at a time can start up using these keys. To handle migration issues
in clouds, we can make use of existing techniques (, [30]), however, it is out
of the scope of this work.

When the Palæmon instance starts up, it attests itself via Intel attestation
service (IAS) [3, 38]. On a successful attestation, it gets a report from IAS
that associates its MRE with its public key. The instance can send this report
to the Palæmon CA to obtain a certificate for the public key mentioned in the
report. Clients that connect to the instance via TLS are served this
certificate after successful verification of the certificate by the instance.
The clients can then verify the instance via TLS by ensuring that the
certificate is signed by the Palæmon CA. Alternatively, clients can request
the IAS report via a REST API provided by Palæmon, and then verify that the
report:  _(i)_ was indeed signed by IAS, and  _(ii)_ associates the Palæmon
MRE with the public key of the certificate.  At that point, the clients know
that the instance runs inside of a TEE and has the correct MRE, , they can now
safely send requests.

Note that clients might themselves run inside a TEE and obtain the permitted
MREs from their security policy. Moreover, they might be limited to connecting
only to certain Palæmon instances identified by their public keys.

###  IV-C Single Instance Enforcement

Each Palæmon instance is identified by a unique public key that it created
during initial setup. Upon restart, the instance will get the same public key
from sealed storage. An attacker might try to start two or more instances with
the same public key and, in this way, may try to roll back some file system
updates of applications. Indeed, an application that stored its _tag_ in one
Palæmon instance might be served an old tag if it is connected to the other
instance after a restart. We prevent this attack by ensuring, as part of the
rollback protection (see below), that at most one instance of Palæmon runs
with a given public key.

###  IV-D Protection against Rollback Attacks

Fig. 6: Rollback protection in Palæmon using monotonic counters.

We use a simple yet effective approach to protect against rollback attacks on
Palæmon’s database. The problems that we need to address are as follows.
First, the “monotonic counters” provided by the SGX platform [15] can be
incremented no more than 20 times per second, , at most every 50 ms. This
means that waiting for a new increment of the counter can cause delays of
about 75 ms, , about 25 ms on average for the current increment to finish in
addition to the 50 ms of the next increment. This delay might cause unwanted
delays in applications since this limits the number of updates to the tags of
a single application to about 13 times per second, which is far too few for
many applications. Moreover, continuous increment of monotonic counters
introduces a wear and tear of the counters, leading to a dramatically reduced
life expectancy of these counters [15].

We therefore adopt an alternative approach based on the observation that
Palæmon runs on well-maintained hosts that have very limited unscheduled
downtimes. For example, there would typically be an uninterruptible power
supply (UPS) system to reduce the likelihood of power outages. For any
unscheduled outage, we expect that we need to perform a fail-over to another
Palæmon service instance anyhow. In our approach, illustrated in Fig. 6, we
protect against rollbacks using a version number $v$ stored in Palæmon’s
encrypted database and a hardware-based monotonic counter $c$ that keeps track
of this version number. Upon startup, Palæmon checks that the monotonic
counter $c$ and the version $v$ of the database match, , $v=c$, and otherwise
exits. Palæmon then increments the monotonic counter before accepting any
request. The database is now trailing the monotonic counter, , $v<c$. This
will prevent any further restarts unless Palæmon updates $v$ during shutdown.
Furthermore, Palæmon checks that the increment effectively yields $c=v+1$.

Any higher value for $c$ would indicate that a second instance is already
running. In such a case, Palæmon would exit immediately. As common for
containers, Palæmon is terminated via a signal. In that case, it shuts down
all connections and stops accepting new requests. Existing requests are still
processed and the internal database is updated. The final step is to increment
the version in the database and shut down the service. In this way, the
monotonic counter and the version of the database agree again, thus allowing
Palæmon to restart.

Note that the monotonic counter $c$ and the version $v$ are not incremented
for every update to the expected tag (see $\S$III-D). Thus, our rollback
protection mechanism can achieve significantly higher throughput and lower
latency compared to previous approaches (see $\S$V). However, in this work, we
treat a systems crash as a case of attacks, , we ensure the consistency and
freshness with an assumption about availability. Ensuring both consistency and
availability is a challenging task, which we currently address in our ongoing
work.

###  IV-E Policy Access Control

Security policies can be accessed via a REST API protected by TLS. Clients
connecting to a Palæmon instance attest the instance by verifying that it has
a certificate issued by the Palæmon CA. A client must also provide a client
certificate, which is stored upon creation of the security policy. All further
accesses (, read, update, delete) to this policy are limited to the clients
with the same certificate, and also require approval by the policy board.
Multiple clients can easily share the same certificate by running as part of a
single security policy.

##  V Evaluation

We evaluate Palæmon with respect to its security and its performance. First,
we introduce a security analysis. Second, we measure the overheads both at the
micro-level in controlled environments and at the macro-level in real
deployments.

###  V-A Security Analysis

In terms of secret management, Palæmon provides stronger security guarantees
compared to previous systems like Barbican [58] or Vault [33]. Palæmon
protects against _eavesdropping on any communication_ by only supporting TLS-
based communication using ciphers with _perfect forward secrecy_ [31]. To
avoid man-in-the-middle attacks by potentially compromised root CAs [65, 36],
we run our own Palæmon CA to generate the Palæmon TLS certificates. This CA
runs inside a TEE and is controlled by a policy board to protect against
malicious software updates. Each Palæmon instance serves its IAS reports as a
second way to be attested by its clients. Clients are free to combine both
approaches.

Palæmon protects the CIF of data by encrypting all data and tracking the
freshness if not otherwise ensured—for all data at rest, in transit or in main
memory. This requires us to effectively protect the secrets, such as the
symmetric and/or asymmetric keys. Palæmon protects against unauthorized
accesses to secrets by enforcing the following:  _(i)_ secrets are defined in
the context of a security policy;  _(ii)_ each application runs in the context
of a single security policy;  _(iii)_ only applications running in the context
of a security policy are permitted to retrieve secrets of this security
policy: the security policy specifies for each of its applications which
secrets it is permitted to access; and  _(iv)_ both the application code and
its file system state are specified in the security policy and attested before
the application can gain access to any secrets.

Access to security policies is controlled with the help of certificates. Each
client has to be authorized to access a security policy: a client must know
the private key that corresponds to the public key used to create the security
policy. Note that no other entity, like the provider managing the Palæmon
instance, can access this policy without knowing this private key. Therefore,
only the client that creates a security policy controls access to this
security policy. Any policy access must additionally be authorized by its
policy board to protect against authorized but Byzantine client accesses.

Palæmon protects the _confidentiality of stored secrets_ by encrypting all
secrets at rest with a randomly selected key only known to itself. By always
executing inside of a TEE, Palæmon protects against _memory analysis of a
running instance_. A Palæmon instance is protected against some types of
_control of its storage backend_ by providing:  _(i)_ protection against
_manipulation_ (including rollbacks), by maintaining a Merkle tree of its
files and storing this hash value in a sealed file (protection against
rollbacks is ensured with the help of a monotonic counter); and  _(ii)_
protection of the _availability and durability of the storage backend_ by the
use of a trusted object storage like Pesos [43].

Fig. 7: Startup times (80 kB binary) are typically not dominated by
measurement overhead as Palæmon only measures code (left bars).

Palæmon protects against _attackers with superuser access_ by executing
securely inside a TEE. All communication, files and data outside the TEE are
always encrypted. Hence, even users with superuser privileges cannot access or
modify any secrets. Access rules are defined and enforced per security policy,
and a security policy can only be modified when an authorized client requests
a change that must then be approved by the policy board of the security
policy.

Although side-channel attacks are out of scope of this work, it is worth to
mention that the underlying SCONE platform can protect against L1-based side
channels attacks [57] and is hardened against Iago attacks [17]. To mitigate
the various variants of Spectre [42], we can use LLVM-extensions, e.g.,
_speculative load hardening_ [13] that prevent exploitable speculation.

###  V-B Micro-benchmarks

Evaluation Settings. All our experiments are executed on a rack-based cluster
of Dell PowerEdge R330 servers. Each machine is equipped with an Intel Xeon
E3-1270 v6 CPU and 64 GB of RAM. The machines are connected to a 20 Gb/s
switched network. SGX is statically configured to reserve 128 MB of RAM for
the enclave page cache (EPC) [20]. We use Ubuntu 16.04 LTS with Linux kernel
v4.13.0-38. The CPUs use the latest microcode patch level.

The underlying SCONE runtime also supports emulation mode (EMU) to run legacy
applications without any TEE support. We use EMU during the evaluation where
indicated to highlight the performance overhead of the TEE.

Enclave Startup Times. First, we evaluate how long it takes to start an
application inside an SGX enclave. Palæmon automatically loads an application
inside of an enclave with the help of a modified loader. Setting up an enclave
includes:  _(i)_ adding pages to the enclave;  _(ii)_ measuring their content;
_(iii)_ evicting pages if the enclave’s size exceeds the EPC; and  _(iv)_
bookkeeping tasks such as allocating memory and copying data.  We measured the
throughput of each component with a micro-benchmark (see Table II).

On new SGX-capable CPUs, the Palæmon runtime will dynamically allocate heap
memory, , startup times are mainly determined by the time it takes to load the
code with some minimum heap. When the runtime fails to allocate memory, it
tries to add new heap pages to the enclave. Current CPUs support SGX enclave
sizes of up to 64 GB. This limit is expected to increase much further, , we
could run most applications inside of enclaves.

To ensure the integrity of an enclave, we need to measure all its code and
initialized data segments. The internal memory allocator is aware of the
position of the contiguous enclave memory. It will not use memory outside of
the enclave, and it overwrites requested memory with zeros, avoiding
measurement of added heap pages. For small initial enclaves—which we expect to
be common when adding heap memory dynamically—bookkeeping and page addition
times are typically the dominant factors, despite the slow measuring speed
(see Fig. 7).

Attestation and Configuration. First, we evaluate how long it takes to attest
and configure an application. The advantage of Palæmon over the traditional
way using IAS to perform attestation is that Palæmon runs on the local
cluster. We measured the time it takes to perform the individual steps of
remote attestation (§IV-B). The IAS experiment ran on servers in Europe and in
Portland, OR, USA (close to IAS servers). In the future, we will support both
IAS and DCAP [64]. Palæmon’s attestation infrastructure will stay the same, as
it attests other factors like the file system state.

Table II: The average throughput of measuring pages is about an order of magnitude slower than evicting or adding pages. Bookkeeping |  Eviction |  Measurement |  Addition
---|---|---|---
1,292 MB/s |  1,219 MB/s |  148 MB/s |  2,853 MB/s
Fig. 8: Attestation and configuration latencies: even when located close to
Intel’s IAS server, attestation with IAS takes about an order of magnitude
longer than with Palæmon.

Fig. 8 shows the time it takes to:  _(i)_ initialize the necessary resources;
_(ii)_ send the quote to Palæmon;  _(iii)_ wait for Palæmon to confirm the
successful attestation; and  _(iv)_ receive the configuration.  The
initialization phase includes key pair generation, DNS resolution, connection
establishment, and TLS handshake with Palæmon. Overall, the initialization
time is similar for each attestation service and is dominated by the TLS
handshake.

Obtaining and sending the quote takes longer for IAS variants for two reasons.
First, performing IAS attestation requires providing information that is
embedded into the generated quote, which adds one round trip. Second, Palæmon
attestation cryptography (Ed25519 [7]) is less expensive than the one used by
IAS (EPID [11]). However, the dominating factor for IAS is the time spent
waiting for the attestation. Palæmon has to verify the quote either by
querying the IAS or by verifying the signature and looking up the public key
of the quoting enclave [20] (QE). Overall, Palæmon attestation takes around 15
ms to complete, which is an order of magnitude faster than IAS attestation
which takes 280 ms when performed from the USA, or 295 ms from Europe.

Palæmon also decouples application startup from IAS. Our benchmark starts
multiple minimal programs in parallel to measure the startup throughput and
latency. Fig. 9 depicts the latency and throughput for different attestation
variants. In the Native case (SGX and attestation are not involved), the
throughput scales well until all eight hyper-threads are fully utilized. At
this point, the system runs around 3,700 programs every second. If the program
is compiled with SGX but without attestation (SGX w/o), the throughput drops
to about 100 executions per second. This variant does not scale well with
increasing parallelism. We tracked down the bottleneck to the Intel SGX driver
synchronising EPC page (de)allocations with a single lock. Since every enclave
has to obtain EPC pages at roughly the same time, this lock basically enforces
page requests to be served sequentially.

Fig. 9: Startup latency and throughput using attestation variants.

With IAS and Palæmon, the startup routine performs remote attestation before
executing the actual program. With Palæmon attestation, we quickly reach the
maximal achievable start rate of about 90 runs per second. IAS attestation
needs a considerable amount of parallelism to partially hide the higher
latency, reaching about 40 runs per second (60 parallel instances) at 1.4 s
latency.

Rollback Protection. Palæmon protects against rollback attacks by ensuring
that the root tags of all volumes of a process are sent to Palæmon on each
file system synchronisation, file closing, as well as on program exits.
Palæmon stores the tags in its encrypted database. We measure the latency of
the Palæmon runtime reading and updating the most recent tag in the Palæmon
service to evaluate the overhead of rollback protection (Fig. 11 left). The
update latency is roughly 6 $\times$ higher than the read latency, as the
Palæmon service database needs to be committed to disk for updates but not for
reads.

Fig. 10: File-based monotonic counters are 5 orders of magnitude faster than
platform counters for the same rollback protection.

Palæmon itself is protected against rollbacks with the help of a monotonic
counter. To that end, we use the monotonic counters [15] provided by the SGX
platform. Independent measurements have shown that these counters allow
between 4 [49] and 17 [8] increments per second. TPM-based counters have a
throughput of approximately 10 increments per second and wear out after 300 k
to 1.4 M writes [66]. The ROTE system [49] stores the monotonic counters in
memory of a group of servers, achieving a throughput of about 500 operations
per second with 4 servers in a local area network. However, the protection of
ROTE against rollbacks is considered to be less robust than when using the
platform counters. In contrast, the anti-rollback protection offered by
Palæmon in _strict mode_ is as safe as the underlying monotonic counters, ,
Palæmon and its applications can only be rolled back if an attacker can roll
back the monotonic counters of the platform.

We measure how fast we can increment a monotonic counter in the following
scenarios (see Fig. 10):  _(a)_ using counters provided by the underlying
platform and the Intel SGX SDK;  _(b)_ by opening a file, incrementing the
integer stored in the file, writing back the new counter value and closing the
file upon exit, when running in native mode;  _(c)_ like _(b)_ but running
inside SGX enclaves without encrypting the file;  _(d)_ like _(c)_ but
encrypting the file transparently with Palæmon; and  _(e)_ like _(d)_ but also
updating the tags with Palæmon by running in _strict mode_ , , the file is
protected against rollbacks.  Applications using Palæmon often do not need to
use monotonic counters since files are rollback-protected. Still, they
sometimes use them to track for instance the number of executions.

Using the platform’s monotonic counters, we reach only 13 increments per
seconds. Note also that this approach requires applications to be rewritten in
order to be protected against replay attacks. Variant _(b)_ shows that we can
reach a much higher throughput of 682 k increments per second when using a
simple file-based counter. When running inside of enclaves, throughput
increases as files are transparently memory-mapped by the SCONE runtime to
counter SGX overhead. When encrypting files, Palæmon automatically performs
caching, increasing the throughput even more. Sending the updated tag to a
Palæmon instance only slightly reduces the throughput. Our measurements show
that applications could use a file-based counter and achieve throughputs that
are 5 orders of magnitude higher than using the counters provided by the
platform. This approach relies on our assumption that system crashes are
considered attacks.

Fig. 11: Left: latency of Palæmon tag reads and updates. Right: reading
overhead for a file with 1 or 10 secrets normalized by the time to read a
plain file. Fig. 12: Latency to retrieve multiple secrets (up to 100) from a
Palæmon service deployed locally, from the same data centre (DC) or from an
instance running on a different continent.

Secret Injection Latency. We measure the impact of injecting secrets in a file
by an application running inside an enclave. To that end, we read a $4$ kB
file in which we inject 1 and 10 secrets (Fig. 11 right). We show the latency
as well as the overhead compared to the baseline on top of each bar. Palæmon
achieves better latencies for files with injected secrets—even compared to the
_plain file_ baseline—because the secrets are injected during startup and stay
in enclave memory.

Secret Access Latency. Palæmon supports the retrieval of keys from remote
Palæmon services. We measure the overhead of retrieving local and remote
secrets, , when using Palæmon in a decentralized fashion (Fig. 12). There is
no visible increase in latency when retrieving 1, 5, 50 and 100 keys of 32
bytes. As a matter of fact, retrieving 50 or 100 keys consistently outperforms
1 or 5 keys. However, there is an impact if a peer service is located on a
different continent instead of the same data centre. This is mainly caused by
the time it takes to establish of a TLS connection.

Fig. 13: Palæmon’s approval service: throughput/latency (left) and response
latency (left) for different geographical deployments (from local to
intercontinental).

Approval Service. We measure the performance of the approval service running
inside a TEE and compare it against a native version. In both variants, we
consider HTTP connections with and without TLS to show its impact. The
approval service and the client issuing requests run on the same rack. We show
the measured throughput/latency plots for these four combinations in Fig. 13
(left). In these experiments, we issue approval requests at fixed rates
(achieved throughput on the horizontal axis) until the response latencies
spike. We observe that the Palæmon runtime over TLS achieves around 210
requests per second before the reply latencies spike. We consider these
results satisfactory for our settings, where policy updates occur at slower
rates.

Next, we investigate the impact on the response latency when the clients of
the approval service are geographically distant. We configure this experiment
with five different distances (see the horizontal axis of Fig. 13 (right)),
from the closest (same rack) to the furthermost deployment with
intercontinental latencies. We show the average latency of Palæmon’s approval
service response with the 95% confidence interval. As expected, the network
latencies dominate the costs, with up to 1.36 s response time in the worst
case.

###  V-C Macro-benchmarks

Our macro-benchmarks run using real-world systems, such as Barbican and Vault
key management systems (KMSs), the NGINX web-server, the Memcached cache
system, the MariaDB database server (a fork of MySQL), and the ZooKeeper
distributed coordination service. Software versions used in macro-benchmarks
are presented in Table I. All these systems benefit from Palæmon for
additional security guarantees; we evaluate its impact on performance.

Fig. 14: _Barbican_. Throughput/latency of several variants (native, Palæmon
and BarbiE [16]) with two different microcodes.

Barbican. We begin by measuring the throughput/latency ratio with different
variants of Barbican (v5.0). Fig. 14 shows:  _(i)_ native using a simple
crypto plugin;  _(ii)_ Palæmonon SGX hardware (HW); and  _(iii)_ BarbiE [16]
(Barbican using SGX SDK v2.0 as HSM).  All variants run on CPython v2.7.14.
Additionally, we perform the Barbican measurements for CPUs with _pre-Spectre_
(version 0x58) as well as _post-Foreshadow_ microcodes (0x8e). Palæmon
exhibits some overhead since the arguments of system calls must be checked by
the _syscall shield_ while arguments are being copied out of the enclave and
return values are copied back in. BarbiE performs better than Barbican native
due to its small trusted computing base (TCB) and more efficient compiled
code, rather than interpreted. Finally, the observed performance drop is of
approximately 30% when using the newer microcode. We attribute this to the
flushing of L1 cache on enclave exit, required to mitigate the L1TF
vulnerability, as also reported by Intel [39] and _Weichbrodt_ [71]. BarbiE
does not suffer as much since it requires less EPC paging and has a low number
of enclave exits, in line with its number of requests per second.

Fig. 15: _Vault._ Throughput/latency of native (with TLS), Palæmon in
emulation and Palæmon in hardware mode.

Vault. We evaluate Vault (v0.8.1) compiled by gccgo in Alpine Linux (Fig. 15).
We use wrk2 [67] to retrieve secrets from Vault by providing it with an
appropriate token. Vault requires a heap of least 1.9 GB to start, , the
enclave is much larger than the EPC, so paging takes place. Our evaluation
shows for instance that, for latencies below 1 s, Palæmon still achieves 61%
of native throughput when running in hardware, and up to 82% when running in
EMU mode.

Fig. 16: _Memcached._ Throughput/latency of native, Palæmon in emulation and
Palæmon in hardware mode.

memcached. We evaluate the impact of Palæmon for running TLS protected
memcached [25]. In particular, Palæmon injects the certificates and private
keys for TLS termination. We use memtier [61] to load and stress memcached.
Fig. 16 shows the measured latency and throughput of the evaluated systems. We
make a comparison between Palæmon and native memcached using stunnel [73] TLS
connections for both systems. With latencies smaller than 3 ms, Palæmon
achieves 59.5% and 65.3% of native throughput with hardware and EMU mode,
respectively.

NGINX. Along the same lines, we use an encrypted NGINX [62] container image
and rely on Palæmon to:  _(i)_ encrypt all the files;  _(ii)_ inject the
certificates; and  _(iii)_ inject private keys used by NGINX for TLS
termination.  The benchmark issues GET requests on 67 kB files (nowadays’
average size of an HTML web page [41]) with the wrk2 tool (see Fig. 17 (a)).
We see that the overhead of SGX alone is less pronounced than that of
encrypting all files. Tuning the caching done by NGINX could improve the
performance when encrypting files. There is little difference between running
in emulation mode and inside of an SGX enclave, since not much paging is
taking place.

Fig. 17: (a) Throughput/latency for GET requests on 67 kB files, in five
variants of nginx. ZooKeeper: read (b) and write (c) operations. (d) MariaDB
with TPC-C benchmark (d): increasing buffer pool helps native more than EMU or
hardware.

ZooKeeper. Next, we evaluate the overhead of Palæmon with the ZooKeeper
coordination service. We deploy a cluster of three nodes and evaluate three
ZooKeeper variants:  _(i)_ native using stunnel [73] for TLS termination
between servers;  _(ii)_ shielded ZooKeeper running together with the JVM in
hardware mode; and  _(iii)_ EMUmode.  We use the ZooKeeper Benchmark [47] to
measure read and write throughput. The read throughput of the shielded
versions is consistently better than the native one (Fig. 17 (b)). The write
throughput (Fig. 17 (c)) exhibits better performances in native mode, as it
involves the execution of consensus [37] via TLS, resulting in more code and
system calls being executed. Our results are on par with _SecureKeeper_ [10],
despite its use of an encryption proxy to protect the content only.

MariaDB. We conclude our macro-benchmarks by measuring the throughput of
MariaDB configured to perform encryption at rest [48]. We use Palæmon to
inject a generated X.509 certificate, the private key and the encryption key.
We execute the TPC-C benchmark [69] and vary the available buffer cache. Fig.
17 (d) presents experimental results. For small buffer pool sizes [59], ,
$<$128 MB, all configurations behave similarly since the main overhead is
hardware I/O. For larger buffer caches, EPC paging increases in hardware mode.
Hence, adding more buffer cache reduces the throughput while it increases the
throughput in emulation and native mode. A fair comparison with the recently
proposed EnclaveDB [60] system is currently not possible since it lacks paging
support and its performance figures are only based on simulations.

##  VI Production Use Case

Finally, we describe a deployment of Palæmon in a real production environment
for a company offering an online service for automatic conversion of
handwritten documents into digital data via machine learning. Typically,
customers of this company want to acquire inference results and, due to the
sensitive nature of the documents, ensure the confidentiality of the input
images. Additionally, the company wants to protect both the inference engine
(implemented in Python) and its machine learning models.

To achieve these security goals, the company has deployed Palæmon as follows.
First, the company relies on SCONE’s file system shields [5] to encrypt Python
code and models used for the inference. The customers use the same mechanism
to encrypt the input images. However, the company and the customers do not
share with each other the keys and tags to decrypt and ensure the freshness of
their digital assets. Instead, they define a dedicate security policy to
define the access control to those. Thereafter, they submit the policy to
Palæmon after performing the attestation (see $\S$IV-B) to ensure code
integrity. To process an image, it takes on average $\leavevmode\nobreak\
323ms$ and $\leavevmode\nobreak\ 1202ms$ ($3.7\times$ slowdown) with the
native and the Palæmon-enabled version, respectively. However, the result is
less than $1.5$ seconds and thus considered acceptable in a production
setting.

##  VII Related Work

Key management systems (KMSs) [58, 53, 33] provide an integrated approach for
generating, managing and distributing cryptographic keys for devices and
applications. They are at the core of secure distributed systems and have been
widely studied. Many approaches rely on cryptographic techniques, often
embedded in secure hardware modules [56, 55].

Recent cloud computing frameworks integrate dedicated services for key
management. _Barbican_ [58] and _Vault_ [33] are popular standalone KMSs. Both
rely on the operating system (OS) for security, and thus consider a weaker
threat model than Palæmon which considers attackers with superuser access (see
§II-A). _Barbican_ and _Vault_ can be protected against such attackers by
running them on top of Palæmon; we evaluate these hardened variants in §V.
Several major cloud providers also offer managed services to create and
control encryption keys, , Amazon [2], Google [29] and Microsoft [51]. Users
must trust the providers to protect their secrets while Palæmon can be both
managed by a provider and attested by users to establish trust in it.

While previous KMSs integrate hardware security modules (HSMs) to provide
better protection, we deem this approach vulnerable to the adversary Palæmon
protects against. An adversary with superuser access can eavesdrop on the HSM
to obtain secrets, or directly hijack the KMS and observe the secrets
distributed to clients.

To the best of our knowledge, while TEEs have been widely used to secure many
applications, only two systems are using TEEs to harden a KMS against
adversaries with superuser privileges. Researchers from Intel proposed the use
of SGX for securing _Barbican_ [16]. Along the same lines, Fortanix’ self-
defending key management service (SDKMS) [44, 6] also uses SGX to securely
generate, store and use cryptographic keys, certificates and various types of
secrets. Both approaches essentially provide a replacement for the
functionality normally provided by HSMs by using enclaves, hence reducing
costs and providing better extensibility. Like classical HSMs, they still need
passwords or personal identification numbers (PINs) in configuration files to
authenticate clients. In contrast, Palæmon provides an integrated approach to
free itself from such sensitive identifiers and use the application code
itself for authentication and authorization. Palæmon also provides several
additional features not found in other systems, , advanced governance by a
policy board, secret sharing between service instances and rollback
protection.

To integrate KMSs into legacy applications without changing their source code,
systems as Vault [33] process configuration files and environment variables
using scripts (, consul-template and envconsul) before executing the
application [21]. However, since this environment is maintained by the OS, it
is thus accessible to attackers with superuser privileges. Hence, this is not
a viable solution to protect against privileged attackers. Palæmon provides
transparency by establishing the environment expected by the application
inside the TEE, hence never exposing its secrets to the OS.

We finally mention that secrets are sometimes stored in configuration
management services _Chubby_ [12], _Consul_ [32], _ZooKeeper_ [37] (or its SGX
version _SecureKeeper_ [10]) along with other configuration data. Note that
ZooKeeper neither encrypts data on disk nor does it protect its network
communication. With Palæmon, we can retrofit the necessary features to protect
ZooKeeper in a cloud context (see §V). It is also worth mentioning that,
Palæmon has been used not only in production, but also in several research
works [68, 45].

##  VIII Conclusion

We introduced Palæmon, a service to manage trust in untrusted environments
with Byzantine stakeholders. Unlike in the Byzantine Fault Tolerance [14]
approach, we can enforce—via remote attestation—that the correct application
code is executed. In this way, we do not need to deploy multiple replicas to
enforce integrity and freshness. Moreover, we also enforce confidentiality. In
order to support application updates, the root of trust of an application is a
group of stakeholders—some of which might be Byzantine. We protect
applications with the help of TEEs: Palæmon clients can securely create
secrets and protect access to these secrets with a security policy, even from
insiders and attackers with superuser access.

To avoid source code changes, Palæmon passes the secrets to applications as
arguments, environment variables and by transparently injecting these into
files. Our evaluation indicates that applications can achieve good throughput
despite running in TEEs. Throughput of our monotonic counters is 5 orders of
magnitude higher than those offered by the SGX platform. We will make Palæmon
available to the research community.

## Acknowledgment

The research leading to these results has received funding from the European
Union’s Horizon 2020 research and innovation programme and by the Swiss State
Secretariat for Education, Research and Innovation (SERI) under the
SecureCloud Project (securecloudproject.eu), grant agreement No 690111, as
well as under the LEGaTO Project (legato-project.eu), grant agreement No
780681.

## References

  * [1] Grant Allen and Mike Owens.  The Definitive Guide to SQLite.  Apress, 2010.
  * [2] Amazon.com, Inc.  AWS Key Management Service.  https://aws.amazon.com/kms/, 2019.
  * [3] Ittai Anati, Shay Gueron, Simon Johnson, and Vincent Scarlata.  Innovative technology for cpu based attestation and sealing.  In Proceedings of the 2nd International Workshop on Hardware and Architectural Support for Security and Privacy, volume 13 of HASP ’13. ACM, 2013.
  * [4] ARM Limited.  Building a secure system using trustzone technology.  White paper, 2009.
  * [5] Sergei Arnautov, Bohdan Trach, Franz Gregor, Thomas Knauth, Andre Martin, Christian Priebe, Joshua Lind, Divya Muthukumaran, Dan O’Keeffe, Mark L. Stillwell, David Goltzsche, Dave Eyers, Rüdiger Kapitza, Peter Pietzuch, and Christof Fetzer.  SCONE: Secure linux containers with intel SGX.  In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’16, pages 689–703. USENIX Association, 2016.
  * [6] Jethro G. Beekman and Donald E. Porter.  Challenges for scaling applications across enclaves.  In Proceedings of the 2nd Workshop on System Software for Trusted Execution, SysTEX’17, pages 8:1–8:2. ACM, 2017.
  * [7] Daniel J. Bernstein, Niels Duif, Tanja Lange, Peter Schwabe, and Bo-Yin Yang.  High-speed high-security signatures.  Journal of Cryptographic Engineering, 2(2):77–89, 2012.
  * [8] Marcus Brandenburger, Christian Cachin, Matthias Lorenz, and Rüdiger Kapitza.  Rollback and forking detection for trusted execution environments using lightweight collective memory.  In 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN ’17), pages 157–168, June 2017.
  * [9] Ferdinand Brasser, Urs Müller, Alexandra Dmitrienko, Kari Kostiainen, Srdjan Capkun, and Ahmad-Reza Sadeghi.  Software grand exposure: SGX cache attacks are practical.  In 11th USENIX Workshop on Offensive Technologies, WOOT ’17. USENIX Association, 2017.
  * [10] Stefan Brenner, Colin Wulf, David Goltzsche, Nico Weichbrodt, Matthias Lorenz, Christof Fetzer, Peter Pietzuch, and Rüdiger Kapitza.  Securekeeper: Confidential zookeeper using intel sgx.  In Proceedings of the 17th International Middleware Conference, Middleware ’16, pages 14:1–14:13. ACM, 2016.
  * [11] E. Brickell and J. Li.  Enhanced privacy id from bilinear pairing for hardware authentication and attestation.  In IEEE Second International Conference on Social Computing, SocialCom ’10, pages 768–775, 2010.
  * [12] Mike Burrows.  The chubby lock service for loosely-coupled distributed systems.  In 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’06), pages 335–350. USENIX Association, 2006.
  * [13] Chanandler Carruth.  Speculative load hardening.  https://llvm.org/docs/SpeculativeLoadHardening.html, 2019.
  * [14] Miguel Castro and Barbara Liskov.  Practical byzantine fault tolerance.  In Proceedings of the Third Symposium on Operating Systems Design and Implementation, OSDI ’99, pages 173–186. USENIX Association, 1999\.
  * [15] Shanwei Cen and Bo Zhang.  Trusted time and monotonic counters with intel software guard extensions platform services.  White paper, Intel Corporation, 2017.
  * [16] Somnath Chakrabarti, Brandon Baker, and Mona Vij.  Intel sgx enabled key manager service with openstack barbican.  arXiv e-prints, 2017.
  * [17] Stephen Checkoway and Hovav Shacham.  Iago attacks: Why the system call api is a bad untrusted rpc interface.  In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’13, pages 253–264. ACM, 2013.
  * [18] Guoxing Chen, Sanchuan Chen, Yuan Xiao, Yinqian Zhang, Zhiqiang Lin, and Ten H. Lai.  Sgxpectre attacks: Stealing intel secrets from sgx enclaves via speculative execution.  arXiv e-prints, 2018.
  * [19] Keen Cook.  The state of kernel self protection project.  In Linux Security Summit, 2016.  https://www.linux.com/videos/state-kernel-self-protection-project-kees-cook-google.
  * [20] Victor Costan and Srinivas Devadas.  Intel sgx explained.  IACR Cryptology ePrint Archive, 2016(086):1–118, 2016.
  * [21] Armon Dadgar.  Why we need dynamic secrets.  https://www.hashicorp.com/blog/why-we-need-dynamic-secrets, 2018\.
  * [22] Christopher Drew and Somini Sengupta.  NSA Leak Puts Focus on System Administrators.  In New York Times, 2013.  https://www.nytimes.com/2013/06/24/technology/nsa-leak-puts-focus-on-system-administrators.html.
  * [23] Zhao-Hui Du, Zhiwei Ying, Zhenke Ma, Yufei Mai, Phoebe Wang, Jesse Liu, and Jesse Fang.  Secure encrypted virtualization is unsecure.  arXiv e-prints, 2017.
  * [24] Emont, Jon and Stevens, Laura and McMillan, Robert.  Amazon Investigates Employees Leaking Data for Bribes.  https://www.wsj.com/articles/amazon-investigates-employees-leaking-data-for-bribes-1537106401, 2018\.
  * [25] Brad Fitzpatrick.  Distributed Caching with Memcached.  Linux Journal, 2004.
  * [26] Ryan Gallagher and Peter Maass.  Inside the NSA’s Secret Efforts to Hunt and Hack System Administrators.  In The Intercept_, 2014.  https://theintercept.com/2014/03/20/inside-nsa-secret-efforts-hunt-hack-system-administrators/.
  * [27] Kenneth Goldman, Ronald Perez, and Reiner Sailer.  Linking remote attestation to secure tunnel endpoints.  In Proceedings of the First ACM Workshop on Scalable Trusted Computing, STC ’06, pages 21–24. ACM, 2006.
  * [28] Dan Goodin.  “Most serious” Linux privilege-escalation bug ever is under active exploit (updated).  In Ars Technica, 2016.  https://arstechnica.com/information-technology/2016/10/most-serious-linux-privilege-escalation-bug-ever-is-under-active-exploit/.
  * [29] Google LLC.  Google Cloud Key Management Service.  https://cloud.google.com/kms/, 2019.
  * [30] Jinyu Gu, Zhichao Hua, Yubin Xia, Haibo Chen, Binyu Zang, Haibing Guan, and Jinming Li.  Secure live migration of sgx enclaves on untrusted cloud.  In Proceedings of tge 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 225–236. IEEE, 2017\.
  * [31] Christoph G. Günther.  An identity-based key-exchange protocol.  In Advances in Cryptology, EUROCRYPT ’89, pages 29–37. Springer Berlin Heidelberg, 1990.
  * [32] HashiCorp.  Consul: Service Discovery and Configuration Management.  https://www.consul.io, 2019.
  * [33] HashiCorp.  Vault.  https://www.vaultproject.io/, 2019.
  * [34] Felicitas Hetzelt and Robert Buhren.  Security analysis of encrypted virtual machines.  In Proceedings of the 13th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE ’17, pages 129–142. ACM, 2017\.
  * [35] Matthew Hoekstra, Reshma Lal, Pradeep Pappachan, Vinay Phegade, and Juan Del Cuvillo.  Using innovative instructions to create trustworthy software solutions.  In Proceedings of the 2nd International Workshop on Hardware and Architectural Support for Security and Privacy, HASP ’13, pages 11:1–11:1. ACM, 2013.
  * [36] L. S. Huang, A. Rice, E. Ellingsen, and C. Jackson.  Analyzing forged ssl certificates in the wild.  In 2014 IEEE Symposium on Security and Privacy, S&P ’14, pages 83–97, 2014.
  * [37] Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed.  Zookeeper: Wait-free coordination for internet-scale systems.  In 2010 USENIX Annual Technical Conference, USENIX ATC ’10. USENIX Association, 2010.
  * [38] Intel Corporation.  Attestation Service for Intel Software Guard Extensions (Intel SGX): API Documentation.  https://software.intel.com/sites/default/files/managed/7e/3b/ias-api-spec.pdf, 2018\.
  * [39] Intel Corporation.  Resources and response to side channel L1TF.  https://www.intel.com/content/www/us/en/architecture-and-technology/l1tf.html, 2018\.
  * [40] Simon Johnson, Vinnie Scarlata, Carlos Rozas, Ernie Brickell, and Frank Mckeen.  Intel Software Guard Extensions: EPID Provisioning and Attestation Services.  In Intel Whitepaper, 2016.  https://software.intel.com/en-us/blogs/2016/03/09/intel-sgx-epid-provisioning-and-attestation-services.
  * [41] KeyCDN.  The growth of web page size.  https://www.keycdn.com/support/the-growth-of-web-page-size/, 2017\.
  * [42] Paul Kocher, Jann Horn, Anders Fogh, , Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, and Yuval Yarom.  Spectre Attacks: Exploiting Speculative Execution.  In 40th IEEE Symposium on Security and Privacy (S&P’19), 2019.
  * [43] Robert Krahn, Bohdan Trach, Anjo Vahldiek-Oberwagner, Thomas Knauth, Pramod Bhatotia, and Christof Fetzer.  Pesos: Policy enhanced secure object store.  In Proceedings of the Thirteenth EuroSys Conference, EuroSys ’18, pages 25:1–25:17. ACM, 2018.
  * [44] Ambuj Kumar, Anand Kashyap, Vinay Phegade, and Jesse Schrater.  Self-Defending Key Management Service with Intel Software Guard Extensions.  White paper, Fortanix, 2018.
  * [45] Do Le Quoc, Franz Gregor, Jatinder Singh, and Christof Fetzer.  SGX-PySpark: Secure distributed data analytics.  In Proceedings of the World Wide Web Conference (WWW), 2019.
  * [46] Sangho Lee, Ming-Wei Shih, Prasun Gera, Taesoo Kim, Hyesoon Kim, and Marcus Peinado.  Inferring fine-grained control flow inside SGX enclaves with branch shadowing.  In 26th USENIX Security Symposium, USENIX Security ’17, pages 557–574. USENIX Association, 2017.
  * [47] Chen Liang, Andrew Ferguson, and Rodrigo Fonseca.  Zookeeper Benchmark.  https://github.com/brownsys/zookeeper-benchmark, 2014.
  * [48] MariaDB.  Data-at-rest encryption.  https://mariadb.com/kb/en/library/data-at-rest-encryption-overview/, 2019\.
  * [49] Sinisa Matetic, Mansoor Ahmed, Kari Kostiainen, Aritra Dhar, David Sommer, Arthur Gervais, Ari Juels, and Srdjan Capkun.  ROTE: Rollback protection for trusted execution.  In 26th USENIX Security Symposium, USENIX Security ’17, pages 1289–1306. USENIX Association, 2017.
  * [50] Nicholas D. Matsakis and Felix S. Klock, II.  The rust language.  In Proceedings of the 2014 ACM SIGAda Annual Conference on High Integrity Language Technology, HILT ’14, pages 103–104. ACM, 2014.
  * [51] Microsoft Corporation.  Microsoft Azure Key Vault.  https://azure.microsoft.com/en-us/services/key-vault/, 2019.
  * [52] Mathias Morbitzer, Manuel Huber, Julian Horsch, and Sascha Wessel.  Severed: Subverting amd’s virtual machine encryption.  In Proceedings of the 11th European Workshop on Systems Security, EuroSec’18, pages 1:1–1:6. ACM, 2018.
  * [53] Kevin Moser, Eli Klein, Joey Geiger, Joshua Timberman, James FitzGibbon, and Thom May.  Chef Vault.  https://github.com/chef/chef-vault, 2019.
  * [54] Talal H. Noor, Quan Z. Sheng, Sherali Zeadally, and Jian Yu.  Trust management of services in cloud environments: Obstacles and solutions.  ACM Comput. Surv., 46(1):12:1–12:30, 2013.
  * [55] OASIS.  OASIS Key Management Interoperability Protocol (KMIP) TC.  https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip, 2018.
  * [56] OASIS.  OASIS PKCS 11 TC.  https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=pkcs11, 2018\.
  * [57] Oleksii Oleksenko, Bohdan Trach, Robert Krahn, Mark Silberstein, and Christof Fetzer.  Varys: Protecting SGX enclaves from practical side-channel attacks.  In 2018 USENIX Annual Technical Conference, USENIX ATC ’18, pages 227–240. USENIX Association, 2018.
  * [58] OpenStack.  Barbican.  https://wiki.openstack.org/wiki/Barbican, 2018.
  * [59] Oracle Corporation.  InnoDB Startup Options and System Variables, MySQL 8.0 Reference Manual.  https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html, 2019\.
  * [60] Christian Priebe, Kapil Vaswani, and Manuel Costa.  EnclaveDB: A Secure Database Using SGX.  In 2018 IEEE Symposium on Security and Privacy, S&P ’18, pages 264–278, 2018.
  * [61] Redis Labs.  NoSQL Redis and Memcached traffic generation and benchmarking tool.  https://github.com/RedisLabs/memtier_benchmark, 2019.
  * [62] Will Reese.  Nginx: The high-performance web server and reverse proxy.  Linux Journal, 2008(173), 2008.
  * [63] Reuters.  Ex-Microsoft employee charged with leaking trade secrets to blogger.  https://www.reuters.com/article/us-microsoft-tradesecret-idUSBREA2J07K20140320, 2014\.
  * [64] Vinnie Scarlata, Simon Johnson, James Beaney, and Piotr Zmijewski.  Supporting Third Party Attestation for Intel SGX with Intel Data Center Attestation Primitives.  White paper, Intel Corporation, 2018.
  * [65] Christopher Soghoian and Sid Stamm.  Certified lies: Detecting and defeating government interception attacks against ssl (short paper).  In Financial Cryptography and Data Security, pages 250–259. Springer Berlin Heidelberg, 2012.
  * [66] Raoul Strackx and Frank Piessens.  Ariadne: A minimal approach to state continuity.  In 25th USENIX Security Symposium, USENIX Security ’16, pages 875–892. USENIX Association, 2016.
  * [67] Gil Tene et al.  wrk2 HTTP benchmarking tool.  https://github.com/giltene/wrk2, 2018.
  * [68] Bohdan Trach, Oleksii Oleksenko, Franz Gregor, Pramod Bhatotia, and Christof Fetzer.  Clemmys: Towards secure remote execution in FaaS.  In Proceedings of the 12th ACM International Conference on Systems and Storage (SYSTOR), 2019.
  * [69] Transaction Processing Performance Council.  TPC Benchmark C.  http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf, 2010\.
  * [70] Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and Raoul Strackx.  Foreshadow: Extracting the keys to the intel SGX kingdom with transient out-of-order execution.  In 27th USENIX Security Symposium, USENIX Security ’18, pages 991–1008. USENIX Association, 2018.  See also technical report Foreshadow-NG [72].
  * [71] Nico Weichbrodt, Pierre-Louis Aublin, and Rüdiger Kapitza.  sgx-perf: A performance analysis tool for intel sgx enclaves.  In Proceedings of the 19th International Middleware Conference, Middleware ’18, pages 201–213. ACM, 2018.
  * [72] Ofir Weisse, Jo Van Bulck, Marina Minkin, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Raoul Strackx, Thomas F. Wenisch, and Yuval Yarom.  Foreshadow-NG: Breaking the virtual memory abstraction with transient out-of-order execution.  Technical report, 2018.  See also USENIX Security paper Foreshadow [70].
  * [73] Wesley Wong.  Stunnel: SSLing Internet Services Easily.  White paper, SANS Institute, 2001.
  * [74] Y. Yu, H. Wang, B. Liu, and G. Yin.  A trusted remote attestation model based on trusted computing.  In 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom ’13, pages 1504–1509, 2013\.
  * [75] Kim Zetter.  NSA Hacker Chief Explains How to Keep Him Out of Your System.  In Wired, 2016.  https://www.wired.com/2016/01/nsa-hacker-chief-explains-how-to-keep-him-out-of-your-system/.