# Empowering Data Centers for Next Generation Trusted Computing

Aritra Dhar† Supraja Sridhara‡ Shweta Shinde‡ Srdjan Capkun‡ Renzo Andri†

†Computing Systems Lab, Huawei Zurich Research Center ‡ETH Zürich

###### Abstract

Modern data centers have grown beyond CPU nodes to provide domain-specific
accelerators such as GPUs and FPGAs to their customers. From a security
standpoint, cloud customers want to protect their data. They are willing to
pay additional costs for trusted execution environments such as enclaves
provided by Intel SGX and AMD SEV. Unfortunately, the customers have to make a
critical choice—either use domain-specific accelerators for speed or use CPU-
based confidential computing solutions.

To bridge this gap, we aim to enable data-center scale confidential computing
that expands across CPUs and accelerators. We argue that having wide-scale
TEE-support for accelerators presents a technically easier solution, but is
far away from being a reality. Instead, our hybrid design provides enclaved
execution guarantees for computation distributed over multiple CPU nodes and
devices with/without TEE support. Our solution scales gracefully in two
dimensions—it can handle a large number of heterogeneous nodes and it can
accommodate TEE-enabled devices as and when they are available in the future.
We observe marginal overheads of $0.42$–$8\%$ on real-world AI data center
workloads that are independent of the number of nodes in the data center. We
add custom TEE support to two accelerators (AI and storage) and integrate it
into our solution, thus demonstrating that it can cater to future TEE devices.

##  1 Introduction

The cloud service providers (CSP) have moved away from CPU-centric monolithic
design where CPU, memory, and devices are on the same physical platform [1].
Instead, domain-specific accelerators or DSAs have become an essential part of
modern data centers [2, 3, 4] for workloads such as AI/ML [5, 6], graph
processing [7, 8], in-memory cache [9, 10], and many more. DSAs are often
deployed as standalone nodes in a rack with network functions [11] to
communicate with other nodes [12, 13]. DSAs are shared between multiple
tenants [14] concurrently to maximize resource utilization and scalability.

The heterogeneity in the modern multi-tenant cloud poses new security
challenges for ensuring the integrity and confidentiality of users’ data and
execution. Specifically, the attack surface includes mutually distrusting co-
tenants as well as the CSP itself. Reasoning about an untrusted CSP requires
protecting against a large attack surface, including all the CSP software and
intermediate devices necessary for running the cloud. More importantly, it
necessitates stopping the CSP from maliciously reconfiguring the nodes being
used by tenants [15].

Trusted execution environments (TEEs) are one of the building blocks to enable
a secure multi-tenant cloud. They provide enclaves to execute user code
without trusting the OS or hypervisors [16, 17, 18, 19]. CPUs [20, 21, 22, 23,
24, 25] as well as on DSAs [26, 27, 28, 29, 30] support hardware-enabled
enclaves. Given their effectiveness, TEEs have become an important offering in
the cloud [31, 32, 33, 34]. Although TEEs serve a necessary role, they are not
sufficient if the user wants to fully use the heterogeneity in a modern multi-
tenant cloud.

Figure 1: Data center topology. Racks are connected over high-speed optical
switches. Inside a rack, multiple nodes of the same type are clustered
together and then connected over InfiniBand which communicates with the data
center-level switch. The management plane (MP) performs resource allocation,
revocation, and monitoring.

Only a small fraction of the data-center nodes that can be rented out to
tenants are TEE-enabled. Consider a tenant who wants to perform a computation
that is best suited for an AI accelerator node. With state-of-the-art
solutions, such a tenant has to choose between performance and security.
Either execute their computation on a TEE-enabled node (e.g., CPU) which is
not the optimal device, thus sacrificing performance. Or execute on the non-
TEE AI accelerator in an unprotected environment. One potential solution to
address this problem is to make all nodes TEE-enabled, but this is not a
practical. For protecting non-TEE nodes, prior works have shown the
feasibility of doing so via bus-level isolation if they are directly
physically connected to a TEE host [35, 36, 37]. Such host-centric solutions
do not apply to a data-center setting where nodes are connected in clusters
and racks as shown in Fig. 1. On the other hand, solutions that enable TEE
abstractions for a single rack containing non-TEE nodes seem promising [38].
However, they do not scale to multiple racks and are not designed to leverage
nodes that have TEE support. Lastly, simply connecting TEE nodes and rack-
level solutions are not sufficient, it does not make them collectively secure.
Doing so requires careful design considerations for security as well as
scalability at a data-center level.

In this paper, we investigate the feasibility of enclaved execution that
encompasses multi-tenant heterogeneous nodes that go beyond just TEE-enabled
CPUs. We design a distributed TEE solution that allows a tenant to securely
use TEE nodes (including CPUs and DSAs) and non-TEE legacy nodes. We use three
main insights to achieve this goal. First, we use TEEs on CPUs and DSAs when
available. Further, we use such TEEs to protect all the data leaving the
corresponding nodes. Second, we employ a centralized security controller (SC)
that shields all the non-TEE nodes. All non-TEE nodes are placed behind our
trusted controller who imparts TEE properties such as attestation, isolation,
and secure channel.

Put together, these two design decisions seem trivial and sufficient to ensure
that execution on each node is protected. But, on a closer look, it leaves
several gaps that need to be addressed. For example, the CSP can reconfigure
nodes such that multiple tenants have overlapping resources or turn off
protection mechanisms while the tenant is executing sensitive computations. A
physical attacker can bypass our security controller, say by directly
connecting non-TEE nodes physically. Put in other words, local protection at
each node is not sufficient, we need to ensure global isolation. Our third
insight addresses these gaps systematically. We start by ensuring that the
initial state of the nodes is attested and cannot be changed thereafter. The
controller checks that the CSP’s resource management decisions do not violate
resource isolation and secure path guarantees. We provide a secure physical
perimeter to a collection of non-TEE nodes to make them equivalent to the TEE
node’s protection against a physical adversary. In summary, we approach the
problem in a distributed fashion by first ensuring local security and then
enforcing global properties to maintain security.

We demonstrate our proposed solution by prototyping its main components. We
showcase the hardware and software changes required to add TEE support to
DSAs. Specifically, we build hardware TEE primitives for a state-of-the-art AI
accelerator and several commercially available SSDs. We synthesize our
hardware design to show that they are feasible with a reasonable area and
power cost. We then demonstrate that our design is compatible with existing
TEE-enabled GPUs and FPGAs. We build a prototype for our controller to show
the feasibility of our end-to-end design for TEE and non-TEE nodes. We
evaluate our system on real-world data center workloads. Our cycle-accurate
simulation shows that the scalability, performance, and compatibility cost of
our solution are within the reach of practical deployment.

On average, our accelerator incurs an overhead of 12% during setup, 1.0141%
during inference runtime, and $4.4$ ms of teardown time. Our synthesized
hardware design only consumes 0.38 $\mu$W ($1.15\times 10^{-7}\%$) 2786 $\mu
m^{2}$ ($0.2\%$) area on the accelerator die. The modification on the SSDs
only incurs an average 16% & 20% overhead for random sequential read-write
benchmarks. Integrating existing GPU and FPGA only incurs $1.89\%$ and
$0.04\%$ overhead respectively. Our design scales with concurrent multi-
tenants and the SC can support 912 concurrent nodes running inference workload
in full bandwidth. In summary, we make the following contributions:

  * $\bullet$

Design. We explore the design space for enclave execution in a heterogeneous
setting of a multi-tenant cloud environment. We show the feasibility of using
TEE and non-TEE nodes to provide enclave security primitives.

  * $\bullet$

TEEs on DSAs. We show the hardware changes necessary for multi-tenant enclaved
execution on DSAs by building it ourselves (AI accelerator and an SSD) and by
re-using prior proposals (GPUs and FPGAs).

  * $\bullet$

Evaluation. We provide simulation results with real-world workloads that show
the scalability and modest performance costs of our solution.

Figure 2: Data center-scale heterogeneous TEE designs. The figure shows five
potential design choices: $S_{0-4}$. Nodes assigned to tenants T1/T2

##  2 Motivation

Setting & Problem Statement. Modern data centers enable _disaggregated_
execution (e.g., dReDBox project [39, 40]) to improve the performance of
computation by using domain-specific accelerators (DSA) such as GPUs, ML/AI
accelerators, storage accelerators [41], and smart NICs [42]. Fig. 1 shows a
modern datacenter topology consisting of racks connected over a fast
interconnect. Each rack has several CPU or DSA nodes where the CPU nodes are
TEE-capable and a DSA node could either be a TEE or a non-TEE (legacy) node.
The resources, e.g., core, cache, memory, etc. on the TEE nodes can be divided
dynamically into smaller units based on demand. We call these units: fungible
device units (FDU) and they are functionally identical to the underlying
nodes. We assume the nodes (CPUs, GPUs, FPGAs, AI accelerators) have
virtualization layers (hypervisor). Therefore both the non-TEE nodes and FDUs
on the TEE nodes can be shared between multiple tenants. The CSP uses a
management plane for resource allocation, revocation, and monitoring [43, 44,
45]. A tenant deploys a job by submitting a manifest to the management plane.
The manifest specifies the size and type of resources the job needs. In this
disaggregated setting, a job can run on multiple different types of FDUs. We
ensure that trusted FDUs are provisioned on both the CPUs and devices as
specified in the manifest.

We investigate the design principles that are building blocks for enabling
trusted computing in multi-tenant heterogeneous data centers. Specifically, we
want to ensure that the programs and the data from the user stay protected
even when the CSP, their infrastructure, and software stack are attacker-
controlled.

Threat Model. We assume enclave code (CPU enclave, device drivers) running on
CPU cores and FDUs (DSA code) are trusted. Therefore, a tenant trusts all code
that they run on all their FDUs for a job. However, co-located enclaves from
different tenants are mutually distrusting. The CSP can deploy non-TEE and TEE
nodes in the data-center. For TEE nodes, we trust the hardware manufacturer
and these nodes have hardware root-of-trust for generating correct attestation
reports and securing keys. We assume that an up-to-date revocation list is
available to determine if an attestation report is from a revoked node.
Additionally, we trust the node’s TEEs software TCB (e.g., the trusted
hypervisor in ARM CCA, trusted security monitor in Keystone) including the
firmware after verifying its integrity.

The attacker can control the host OS’s, hypervisors, the management plane, and
other applications running on co-located FDUs. The attacker also controls all
nodes and can plug in and out malicious nodes and network devices (e.g.,
switches, routers, network interconnects, etc.). The chipset and BIOS on the
motherboards are also attacker-controlled, making PCIe link encryption and
attestation mechanisms [46] untrustworthy. We assume a physical attacker who
can probe the bus and manipulate all the network traffic.

Finally, we leave Iago [47], side-channels, hardware trojans, and denial-of-
service out of scope for this paper.

##  3 Distributed TEE Design Space Exploration

###  3.1 Potential Designs

We explore five potential ways ($S_{0-4}$) to design a data center-scale
trusted computing infrastructure, as shown in Fig. 2.

$S_{0}$: Centralized SC between non-TEE CPUs. We assume that none of the
nodes, CPUs or DSAs, are TEE enabled. Each DSA is physically connected to a
CPU node and they both are allocated entirely to a single tenant. In such a
topology, any tenant can tamper any node’s memory or the communication
channels. We employ a simple solution where all CPUs are physically connected
to a central trusted security controller (SC). The SC is responsible for
resource isolation and does so by monitoring all communication between CPUs as
well as with remote users. For local access, the SC ensures that nodes of the
same tenant can access each other. To protect the communication channel
between the tenant and the nodes, the SC provides a secure channel. Such an SC
can be implemented in different ways and can be a dedicated hardware module
[38]. This design does not allow multi-tenant devices as nodes are assigned
exclusively to a single tenant. As an SC can be physically connected to a
limited number of nodes, scaling the design beyond a single rack requires
reasoning about global inter-SC co-ordination in the presence of untrusted
nodes. Lastly, the SC has to be aware of all allocation decisions in order to
enforce isolation.

$S_{1}$: Centralized SC between CPU TEEs and DSA nodes. We assume that the CPU
nodes have TEE and can be virtualized. However, the DSA nodes do not have TEE
support. In such a topology, we choose to connect the DSA nodes directly to
the SC, instead of being connected to the CPU as in $S_{0}$. $S_{1}$ scales
better because DSAs can communicate directly, without going via CPUs. Each
unit on a virtualized CPU can be assigned to dedicated DSAs, thus improving
multi-tenancy. To ensure isolation, the SC intercepts all the traffic from the
user and between the nodes and performs access control such that tenants
cannot access each other’s devices. If the tenant communicates directly with a
DSA, the SC provides a secure channel to ensure that the incoming and outgoing
data is protected. However, this makes the SC a bottleneck without improving
the scale beyond a single rack—inter-SC protection still requires a global
view while being aware of all allocation decisions.

$S_{2}$: Semi-centralized SC with TEE and non-TEE nodes. TEE support goes
beyond just CPU nodes as shown in recent proposals for peripheral TEEs [26,
29, 28, 27]. For nodes that are TEE-enabled, we can support the secure
isolation of tenants without requiring the SC. Specifically, we can shift the
SC’s access control checks to local monitors on the TEEs. However, all non-TEE
node communications have to always go through the SC. Although better than
$S_{1}$ and $S_{2}$, the SC still needs a global resource allocation view and
handles communication with the user and non-TEE nodes.

$S_{3}$: Decentralized TEEs. We can eliminate the SC by requiring that all
nodes are TEE-enabled. Such nodes enable fully decentralized TEE, where they
can simply set up secure channels between the nodes after attestation [48,
49]. Further, the user can directly communicate with the node, therefore,
eliminating any potential bottlenecks. Such a decentralized design allows the
TEE nodes to be securely virtualized and be allocated to tenants without the
need for a trusted SC, making it suitable for modern multi-tenant data
centers. However, it has a strong requirement that all nodes have hardware TEE
support, making it infeasible for many devices including legacy DSAs.

$S_{4}$: Hybrid. We need a design that accommodates upcoming TEE-capable DSAs,
existing TEEs, legacy DSAs, and devices without hardware TEE capability.
Noting from $S_{3}$, TEE-capable devices can be completely decentralized
without the need for an SC. Therefore, such nodes can leverage virtualization
and scale across many racks. Noting from $S_{1}$, we can use the SC to protect
non-TEE devices. We can carefully allocate non-TEE devices, such that they are
all behind an SC via a physical connection. We can use the SC to mediate all
communication between non-TEE nodes for access control, in the same fashion as
in $S_{1}$.

Thus, SC acts as the security monitor that provides a secure channel and
filters the traffic to the non-TEE nodes based on the access policies. In
other words, it performs the TEE protection on the behalf of the non-TEE
nodes. Lastly, the SCs across multiple racks only have to keep track of local
non-TEE nodes. Thus, such a design provides the most security and
functionality suitable for multi-tenant data centers.

###  3.2 Security Considerations

Although intuitively $S_{4}$ seems to be the optimal design choice, reasoning
about its security requires careful consideration.

Challenges. While monitoring data on the bus and secure communication between
nodes is necessary it is insufficient to determine the local node’s
configuration state. First, modern DSAs are highly configurable, i.e., the CSP
or hypervisor can push configuration to the DSAs’ firmware. For example,
hypervisors allow pass-through access to peripherals, and can reconfigure
their partition size [15]. NVIDIA A100 GPUs [50] allow their GPU partitions
(known as multi-instance GPU or MIG) to reconfigure partition size. Second,
even if we ensure that the firmware and configuration of the device are
correct after bootup, an attacker-controlled OS/hypervisors can reconfigure a
device or flash its firmware later. So, it is crucial to ensure that the
device initialized from a safe state stays in a safe state. This is necessary
for both TEE and non-TEE-capable devices. Thus, the user or the SC needs to
ensure that the integrity of the device firmware is maintained before and
after secret provisioning. Third, the SC is not aware of the execution of the
DSAs due to their asynchronous nature. Therefore, misbehaving programs on
rogue devices can attempt to bypass the SC and directly access other tenants’
data. Thus, it is necessary to ensure that all communications between nodes
are mediated by either the TEEs or the SC. Finally, we have to protect the SC,
TEE, and non-TEE nodes, and all the communication channels from a physical
adversary.

Insights. We can use secure/measured boot to ensure that both the TEE and non-
TEE-capable devices’ firmware and configuration are correct. The SC must check
that integrity of the all device firmware is maintained and should relay this
to the user before provisioning secrets. We can use attestation to ensure that
the DSAs are running the user-provided code (also known as the device
kernels). TEE-level isolation ensures that the attacker cannot access the
memory of other tenants on the same node. For non-TEE devices, the SC knows
the resource allocation per tenant. At runtime, it ensures tenant isolation by
performing bus-level access control which checks if the accessor is authorized
to access the resource. The communication channel between the nodes is
protected, i.e., an attacker cannot manipulate the data on the bus. All DSAs
require secure and persistent secure on-device key storage to execute remote
attestation and secure boot.

Physical Attacker. We assume that CPU-TEE nodes have hardware protection such
as MEE in SGX, to defend against a physical attacker. We assume that that the
memory on a TEE-DSA node (e.g., Graviton [26], SheF [29]), is part of the chip
that an attacker cannot infiltrate, such as—high performance 3D stacked
memories [51] for accelerators [52], GPU [53], and SoCs [54]. For off-core
memory, memory encryption and integrity protection proposals on GPU [55, 56],
NPU [57, 58], and data center accelerators [59] can thwart physical attackers
with some performance penalties. However, protecting non-TEE nodes is a
significant challenge as no such protection mechanisms are available for
legacy nodes. We assume that the SC and TEE nodes are secured in a physical
container that can detect and prevent tampering [60, 61]. Such mechanisms are
used in existing proposals [38]. Other communications channels between TEE
nodes, TEE nodes & SC, and between SCs are integrity and confidentiality
protected.

TCB. The TCB includes the following: hardware components such as the CPU, and
DSA-SoCs are trusted. The SC software and hardware, and device firmware are
trusted.

###  3.3 Design Requirements

Our solution builds on top of existing data center architecture by keeping the
cloud service provider’s management plane (CSP-MP) unchanged. We assume that
MP, software stack (OS, hypervisors, other applications), and data center
infrastructure (NIC, switches, interfaces) are attacker-controlled. The CSP-MP
is responsible for resource allocation/deallocation, load balancing, workload
deployment, scaling, and management. All nodes have a hardware root of trust
(RoT) that can be used for securely bootstrapping the node. For TEE-capable
devices, this RoT is used for remote attestation to ensure the integrity of
the firmware and the FDUs before the remote users provision secrets. On the
TEE-capable nodes, the security monitor or SM, a high-privileged trusted
entity on the device, provides attestation and enforces isolation between
FDUs. The SM can split the nodes into several multi-tenant units (FDU).

For non-TEE devices, we need external trusted hardware, such as a security
controller (SC), in the rack. The SC is a low-TCB hardened trusted entity that
can be verified and mediates the trusted path from a tenant to a device. Users
provide a manifest that describes the resource requirements, as shown in Fig.
7 in Sec. A.1. Note that our approach assumes that the programmer provides a
static maximum memory size required for each FDU. Once the resources (TEE and
non-TEE) are allocated and the hypervisors are made aware of it, we have to
identify and set up MMIO as well as DMA regions and isolate them. Further, to
ensure that all MMIO and DMA transactions are confidentiality and integrity
protected between nodes, we have to establish a secure channel between FDUs.

##  4 Design

Our design secures both non-TEE and TEE-capable nodes. We ensure that a CPU
node is always TEE-enabled as it runs a attacker-controlled OS or hypervisor.
Additionally, our design requires that the user allocates at least one CPU-TEE
node. We do not allow a non-TEE node to be multi-tenant. We assume that memory
allocation is static and can be decided when an enclave is set up. We observe
in a majority of DSAs, the size of the programs (known as the kernels) and
their memory allocations are predefined in the programs. The DSAs’ programming
model does not allow new memory allocation (e.g., malloc) while inside the
kernel. For other devices such as storage or memory, the user can define the
size based on the workload requirements. Therefore, a programmer can
statically program the physical memory sizes of each remote FDU into the
enclave.

Figure 3: Security Controller (SC) enforces access control between TEE & non-
TEE nodes, and within non-TEE nodes.

Setup. The user submits the number, size, and type of each FDU and non-TEE
node in a manifest file (example in Sec. A.1) along with the code to be
executed on each FDU and non-TEE node to the CSP-MP. The CSP-MP allocates the
requested resources and collect attestation reports from them. The user gets
the attestation reports and verifies them. Upon successful verification, the
user sets up a secure channels with all the resources and provisions the job-
specific secret key ($\mathcal{K}$) to start the computation. We now discuss
how to secure both non-TEE and TEE nodes, and how to compose a rack with both
types of nodes.

###  4.1 Securing Non-TEE Nodes with SC

Security Controller (SC). It is a device with hardware root-of-trust that can
perform memory isolation for itself. The SC acts as the TEE proxy for the non-
TEE nodes and provides TEE properties such as isolation and trusted path for
them. SC ensures that a non-TEE node is allocated to a single tenant and is
not being shared with any other tenant or entity (e.g., CSP). SC protects all
communications between TEE and non-TEE memory by performing authenticated
encryption and decryption using the shared secret ($\mathcal{K}$) provisioned
during user setup. For communication between non-TEE nodes the SC does not
perform authenticated encryption; instead it enforces access control.
Specifically, it checks if source and destination for a transaction belong to
the same job and if so allows it; otherwise it blocks it.

The SC is connected physically (over PCIe) to the non-TEE nodes. During start-
up, the SC first resets the non-TEE nodes to its factory configuration and
creates locally unique identities (e.g., PCIe physical ids which cant be
forged) for them. The SC maintains input and output buffers for the non-TEE
nodes. Since the data in these buffers are in plain text, the SC needs to
enforce isolated access to them. Specifically, it allocates non-overlapping
regions of the buffers to individual non-TEE nodes as shown in Fig. 3. This
ensures that non-TEE nodes cannot access each other’s plain text data. To
enforce access control between non-TEE nodes, the SC maintains a special data
structure called an Enclave Routing Table (ERT). The ERT keeps track of the
nodes that are a part of the same job. The SC adds entries to the ERT when one
of the nodes it is guarding are allocated as part of a new job; the SC deletes
the entry when the job is over.

Setting up and reserving non-TEE node. Given a manifest, CSP-MP asks for non-
TEE node(s) from SC. The SC first checks that the nodes are not part of any
other job by scanning the ERT. If the node(s) are unallocated, the SC starts
the binding process. Specifically, it sends attestation reports for itself and
the non-TEE nodes allocated to the job. If successful, the user provisions a
job-specific secret key $\mathcal{K}$ to the SC. The SC then adds
$\mathcal{K}$ and the nodes associated with the job to the ERT.

Communication: TEE-FDU and Non-TEE node. The SC provides staging buffers where
data can be encrypted and decrypted. For any data leaving the SC, the nodes
move it to SC’s buffer where it is encrypted with the destination’s key before
being sent out. Similarly, any data that is entering the SC (say, to one of
the non-TEE nodes) arrives in the nodes memory mapped in the SC where it is
decrypted with the destinations key. Fig. 3 shows the details of this
mechanism.

Communication: Non-TEE under behind SC. We do not employ authenticated
encryption for data being exchanged between non-TEE nodes behind the same SC.
Instead we use the SC to perform access control using the ERT. Nodes can
communicate using the SC as in intermediary for performing data copies via an
isolated staging buffer in the SC. Alternatively, nodes can setup a zero-copy
mechanism such that they can access each others memory directly. However, the
SC has to mediate such a setup to ensure that only the nodes that belong to
the same job can setup shared memory. Once setup, the SC need not be involved.

Communication: Non-TEE nodes behind different SCs. If a job requires nodes
that are distributed across multiple SCs, the local SCs communicate over a
secure channel using the job-specific key $\mathcal{K}$ to encrypt and decrypt
the data. Since both SCs receive $\mathcal{K}$ during the setup and have valid
ERT entries, this is feasible. Note that SCs do not have to synchronize ERT
entries or key material.

Releasing node. When the tenant initiates a job termination, its primary CPU
FDU sends a termination command to the SC. Since the SC knows that the command
came for a particular job, it resets the non-TEE nodes involved in the job.
Specifically, its clears the internal state and removes the mapping from the
ERT. Once reset, the SC notifies the MP that the node is available for future
allocations.

###  4.2 Securing TEE Nodes

Unlike non-TEE nodes, TEE-nodes do not depend on an SC. TEE nodes provide FDUs
that are set up by the local SM and allows isolation of tenants by leveraging
the on-device TEE primitives. Locally, the SM makes sure that the FDUs on the
same node cannot access each other’s memory. We use hardware root of trust and
remote attestation of TEE nodes to ensure that such an isolation is
trustworthy. Globally, we use authenticated encryption to protect data leaving
the FDU. When two FDUs are explicitly communicating with each other, they can
use a secure channel as shown in prior work [26]. However, in our setting, we
have to handle implicit communication such as DMA and MMIO.

Secure Implicit Communication. On non-enclave platforms, the application
communicates with a device (e.g., GPU) by writing to IO mapped regions,
changes to which are reflected to the device. Alternatively, to send large
amounts of data, the application can setup DMA transfers such that the device
can read application memory directly, without involving the CPU. In our
solution, we have to protect the data when it is being transferred from the
application to the device over an insecure network. We achieve this by
ensuring that all data that leave’s the application enclaves memory is
protected with authenticated encryption. This requires monitoring all memory
regions that are eventually accessed remotely. One way to achieve this is to
expect that the application will perform encryption decryption of data before
it is exposed to remote access. This burdens developer effort and requires
invasive application changes. Even with such changes, existing CPU TEEs do not
allow non-enclave code (local or remote) to access enclave’s private memory.
Therefore, the enclave has to use untrusted address space for communication.
To address these challenges, we enable transparent authenticated encryption
and copying (to non-enclave memory) of all enclave data that is being remotely
accessed.

When applications use DMA or RDMA, they explicitly mark memory for DMA
operations and initiate a data transfers. We can hook such events using a thin
device driver inside the enclave. This way, we can delegate the monitoring of
events and subsequent encryption and data copies to the driver, without
changing the application.

It is more challenging to support MMIO. Applications maps the device memory
into their own virtual address space and from then on simply use load and
store operations, which are expected to reflect into device’s memory. Since
there are no explicit MMIO events or syncs, we cannot hook such memory
operations to perform encryption and copy the data outside the enclave. We
observe that the device’s MMIO-addressable memory range is fixed at boot. When
the application sets up MMIO with the device, it has to map the devices MMIO
address range into the applications address space. Since this setup is
explicit, we can introduce a thin driver that hook the applications MMIO setup
calls. Specifically, it marks the MMIO-mapped pages are inaccessible. This
way, when the application performs MMIO accesses, it results in page faults.
Our device driver can then register a handler for such page faults. During
execution, it can perform encryption-decryption and data copies between
enclave and non-enclave memory region.

##  5 Case-studies

We identify four DSAs that are common across a number of public clouds: GPU,
FPGA, AI accelerator, and SSD. We use the existing TEE proposals—Graviton [26]
for GPU and ShEF [29] for FPGA. We add TEE support to an AI accelerator and
SSD. Specifically, we identify three main hardware components to enable TEE
support. First, an access control unit (ACU) enforces isolation across FDUs on
the same device by performing access checks on shared resources (e.g.,
memory). On each resource access, the ACU knows the current accessor and
performs a look-up to ensure that it has the right permissions. Second, as a
counterpart to the ACU, we maintain an FDU mapping table (FMT) that keeps the
mapping between each FDU on the device to which the job is currently
allocated. The number of table entries is fixed depending on the number of
FDUs supported by the device. These table entries are populated and deleted by
the device SM when an FDU is assigned or removed from a job. Finally, a memory
protection engine (MPE) that performs transparent memory encryption and
integrity protection for all the data entering and leaving the device memory.
We now explain the use of these components in peripheral TEEs.

Existing Device TEEs. Graviton [26] instantiates a TEE for NVIDIA GPUs for
isolated execution of kernels. It implements an ACU in a trusted security
monitor on the GPU, which ensures isolation by performing access checks on all
commands submitted to the GPU from the host as well as memory accesses from
the GPU cores. Graviton tags all GPU pages with job IDs and performs filtering
during runtime. Thus, it does not maintain a separate FMT. Each tenant uses a
dedicated GPU kernel that acts as MPE and uses an authenticated encryption
scheme. Graviton expects a hardware root of trust, such that the security
monitor can generate attestation reports after a secure boot.

ShEF does not allow multi-tenancy on the FPGA but instead provisions the
entire FPGA to a single job. Thus, it does not need an ACU or a FMT. ShEF adds
an MPE called Shield to the user bitstreams which performs encryption and
decryption of all data leaving the tenant’s bitstream. SheF requires a
hardware root-of-trust on the FPGA, such that a trusted security monitor can
perform a secure boot and generate attestation reports. The tenant performs
remote attestation and sets up a secure channel with the FPGA. During setup,
ShEF guarantees the confidentiality and integrity of tenant’s bitstreams.
Summaries of Graviton and ShEF are in Sec encircle. A.4.2 and A.4.1.

Figure 4: SSD Case study. A high-level model of an SSD. Additional hardware
modules isolated enclave access are highlighted.

###  5.1 Adding TEE Support for SSDs

Tenants that share the same SSD can use tenant-specific keys to encrypt their
data before it reaches the SSD, thus removing the need for a TEE. While
straightforward, it is incompatible with modern data center deployments where
storage computations are offloaded to the SSD accelerators, e.g., smart SSDs
[62]. These SSD accelerators require plain text access to the data.
Additionally, recent research shows that a workloads such as data analytic
[63], database query processing [64, 65], AI/ML[66], map-reduce[67], and many
commercial solutions [68, 69] take advantage of in-storage computing. From
these observations, we see that TEE primitives are necessary to support multi-
tenancy on SSDs.

A primer on SSD Architecture. Fig. 4 shows a simplified version of the base
model that contains the host interface layer (HIL) which implements an
interface such as a PCIe subsystem to talk to the CPU, a SSD controller, and a
CPU core (typically an ARM-M profile core) that runs the firmware of the SSD
controller. The SSD controller bridges the HIL with the storage plane. For
more details, refer to Sec. A.6.

Programming Model. SSDs expose four main block-level IO interfaces: read,
write, flush, and trim, while more operations are supported at the file-system
level. As our SSD-TEE works at the block level, the user can format a specific
namespace with any file system, removing the need for application/OS changes.
The existing device driver translates file operations(e.g., read, write,
append) to block-level commands.

TEE Support. We introduce hardware ACU and MPE between the HIL and the SSD
controller. This layer uses an FMT programmed by the Secure Monitor, that runs
on the on-board ARM core, to enforce isolation. On startup, the cloud provide
divides the SSD into different FDUs. The SM creates entries for each of the
FDUs in the FMT to register them. When an FDU is assigned to a job, the SM
looks up the FMT, ensures that the FDU is not allocated to any other job, and
performs Remote Attestation using the SSD’s hardware root-of-trust with the
tenant. Using the RA process, the SM obtains a job-specific key that it
programs into the FMT. During runtime, the ACU intercepts all memory
transactions sent from the HIL to the SSD controller and uses the FMT to
allow/deny transactions. Then, an MPE uses the keys from the FMT to
encrypt/decrypt the data.

Figure 5: AI Accelerator Case study. An overview of the AI accelerator based
on DaVinci [70, 71] architecture. The additional hardware modules to enable
TEE are highlighted.

###  5.2 Adding TEE Support for AI Accelerators

Programming model. AI accelerator applications typically have a host
counterpart that allocates accelerator memory for the computation. The host
application copies data and commands to the AI accelerator using DMA before
starting the execution. After completing execution, the accelerator writes
back the results via DMA to the host memory.

A primer to an AI accelerator. Fig. 5 shows a simplified base model of AI
accelerator with $n$ cores based on the DaVinci architecture [70]. Each core
is capable of running CNN layers and exposes a custom ISA. The cores are
connected to DDR via the broadcast unit (BU). The BU collects memory requests
from the cores and responses from the memory subsystem and routes them to the
correct endpoint. Further, each core has a front end that acts as an interface
for communication with the other cores and BU. A detailed design is in sec.
A.5.

TEE Support. At startup, the cloud provider divided the cores into multiple
FDUs. When the FDU is assigned to a job, the SM ensures that it is not already
allocated by looking up the FMT. Then, it uses the hardware RoT to perform
remote attestation with the tenant to set up a job-specific shared key. The SM
programs this key into the FMT. Because a job can span over multiple AI
accelerator cores, it is necessary to enable isolated inter-core
communication. For this, we use an ACUcore that intercepts all inter-core
traffic and uses the FMT to allow only the cores that belong to the same job
to communicate with each other. Next, to isolate accesses to the DDR memory we
use an ACUmem to filter all accesses based on the FMT. Then, to encrypt all
data leaving the accelerator through the host interface we use an MPE that
performs authenticated encryption using keys from the FMT. The secure
deallocator (dealloc) tracks all tensors allocated to an FDU’s memory space.
When the SM wants to deallocate an FDU, the deallocator issues memory write
commands to zero out the FDU reserved memory space on the DDR.

##  6 Security Analysis

Attacks via untrusted Management Plane (CSP-MP). The untrusted CSP-MP can
violate the job manifest (e.g., allocate wrong resources). However, the remote
verifier can detect such discrepancies via the attestation report from FDUs
and SC (for non-TEE nodes). If CSP-MP allocates revoked nodes or nodes with
old (possibly compromised) firmware, it will detected by remote attestation.
Nodes’ local memory isolation ensures that the CSP-MP or hypervisor cannot
access (read/write/clone) the enclave’s private memory. The CSP-MP cannot
compromise the secure channel between nodes and the remote verifier (both DMA
and RDMA). The CSP-MP cannot allocate the same FDU to two different tenants as
the remote attestation locks a FDU to a single tenant by the shared secret
derived from the remote attestation. The ERT on the SC prevents a non-TEE node
from being allocated to two jobs simultaneously.

Attacks through malicious configurations. Untrusted hypervisors or CSP can
push malicious configurations to TEE or non-TEE nodes. For TEE nodes, the
remote verifier can verify the SM configurations and their integrity via the
attestation report. For non-TEE nodes, the SC reset them to their safe factory
configuration before allocating them to a FDU. The attacker can always reset
non-TEE nodes, but this will erase all the job-specific data and execution
state without leaking secrets.

Attacks via untrusted FDUs/SM. The CSP can add compromised physical devices on
the path (e.g., network devices without SMs) that can intercept traffic and
trigger rogue DMAs and inject interrupts into nodes. This will not leak
information or compromise the execution as the data is confidentiality and
integrity-protected when it leaves a trusted FDU. Remote attestation will
detect the nodes running untrusted SMs (e.g., old vulnerable version, CSP
modifies SM to generate insecure keys, SMs launch modified enclaves) and fail.

Untrusted co-tenants. The node TEE’s memory isolation prevents co-tenants from
accessing a victim tenant’s protected enclave private memory. A tenant can
attempt to perform MMIO or trigger DMA to devices that are part of the
victim’s job. Both these attacks are rendered harmless by encryption.
Alternatively, the attacker can inject software-generated interrupts to the
cores that are running the victim enclave. This could trick the enclave into
starting a DMA with the attacker. This cannot compromise the victim because
all data leaving the enclave’s memory is encrypted and integrity protected.

Attacks via non-TEE nodes. Malicious non-TEE nodes can try to access SC’s
buffers that store other node’s data in plain text, say by reading the in/out
buffers. The SC’s access checks prevents such attacks. The nodes behind the
same SC can try to communicate with each other directly (via MMIO or DMA),
bypassing the SC. Since the nodes are only physically connected to the SC,
such attacks would have to go through the SC’s buses. SC’s bus-level access
control checks prevent such unauthorized communication. We store all keys used
to encrypt data leaving the non-TEE nodes in the SC’s ERT. Malicious non-TEE
nodes can try to read out the ERT, tamper with the ERT, or compromise the SC’s
execution. This is not possible as the SC runs in a privileged execution mode
is inaccessible to other processes. Further, its data and memory is protected
and is inaccessible to all other entities. The non-TEE nodes can try to send
malicious MMIO and DMA requests to non-TEE nodes behind other SCs or other TEE
FDUs. This does not compromise the confidentiality and integrity as all data
is protected with authenticated encryption.

Attacks via untrusted hypervisors. The hypervisors are responsible for the
node’s memory management, loading the enclave’s initial code, and setting up
its memory regions. During remote attestation, we detect tampering of enclave
code or additional initialization data and the verifier will abort. If the
hypervisor allocates enclave memory in the non-secure regions, the local SM
will fail to launch the enclave. The local memory protection of the TEE
ensures that the hypervisor cannot access the enclave’s private memory region
once it starts. SM clears residual states from the memory if the hypervisor
terminates an enclave, making sure that the previous data is not accessible to
the hypervisor or other enclaves. The hypervisor can page out arbitrary
enclave memory. However, TEE’s memory isolation mechanisms ensure that the
enclave pages are inaccessible to the hypervisor. Similarly, the TEE protect
the enclave page table. This prevents the hypervisor from remapping the
enclave’s memory to non-protected memory.

The hypervisor cannot inject DMAs to FDUs as the channel is confidentiality
and integrity protected and will be detected by the trusted FDU. Overlapping
DMA regions does not compromise security as each pair of CPU FDU and DSA FDU
(or SC) uses a unique job-specific key. The hypervisor cannot reply to old
attestation reports as the attestation challenge from the remote verifier
ensures freshness. The hypervisor can inject fake interrupts from devices to
trigger a data transfer from the enclave’s private memory to the device.
However, encrypted DMA ensures no data is leaked.

Physical Attacker. Such an attacker cannot probe non-TEE nodes as they are
confined in a tamper-resilient container; communication between non-TEE nodes
connected to the SC never leaves the SC boundary. The TEE node’s memory is
protected by local TEE features such as memory encryption and integrity
protection. A physical attacker cannot see or modify DMA authenticated and
encrypted data between TEE nodes. Communication between non-TEE nodes from
different SCs is relayed through their local SCs that use authenticated
encryption to protect the data.

##  7 Implementation

We implement the TEE support to AI accelerator and SSD devices in two ways:
RTL and high-level language. We implement the ACU, and FMT in RTL as described
in Sec. 5 and then estimate the area, power, and timing costs.

Functionality. We augment an existing simulation of the device written in a
high-level language (e.g., Python/C++), with ACU, and FMT. For the MPE, we
implement the cryptographic operations in a high-level language , and measure
MPE’s execution time on the on-device ARM-v8 core. For functional correctness,
we ensure that the outputs from the original code with our modified code are
same. Also, we use the functional model to collect execution traces that
capture memory accesses.

Timing. In each case, we use a device-specific cycle-accurate simulator timing
estimation. In such a simulator, we integrate the timing behavior of our
synthesized RTL logic into a known statistical model that computes the overall
timing of the baseline RTL augmented with our custom RTL. We independently
estimate the timing of individual operations from our RTL logic. We then
update the simulator’s timing model to consider these modified timings. Given
such a timing model and the execution traces, we estimate the overall
execution time it would take to perform the same computation on a TEE-
protected device.

For both devices, we implement a remote attestation unit using a HSM chip [72,
73] connected to the simulation platform over $I^{2}C$.

###  7.1 Adding TEE Support: AI Accelerator

We implement ACUcore, ACUmem, FMT, and enclave deallocator on each of the AI
cores’s front-end explained in Sec. 5.2 as dedicated hardware units and the
MPE in SystemVerilog and Python. We add $200$ LoC in SystemVerilog for RTL for
timing simulation and $650$ LoC in Python and shell script to existing
$\sim$15K LoC for functional correctness. We use accelerator’s onboard ARM
core for the authenticated encryption of the MPE.

Simulating the base model. We use a cycle-accurate, event-based simulator
designed for Nvidia’s cycle-accurate GPU that provides accurate timing and
functional correctness [74]. It models timing behavior, data movement (core,
cache, and memory subsystem), and communication [75]. We use the timing model
of an existing AI accelerator (Ascend’s DaVinci AI core [70]). To measure the
timing impact of our RTL logic, we integrate it Nvidia’s statistical model and
event-based simulation. We configure it for 800 MHz, and 150 AI core cycles
latency to memory subsystem. These are standard settings typically used for
accurate power and area estimations [76]. For verifying the correctness of
calculations, we compare the execution output with real execution on a CPU.

We synthesize our SystemVerilog RTL implementation on a high-k metal gate
(HKMG) 28 nm CMOS technology, corresponding to a multi-VT standard cell
library (supply voltage of 0.8 V, estimated in typical-typical (TT) corner
[77]).

DRAM. Computation on the AI accelerator involves execution on the core as well
as interactions with DDR memory. The host copies data to accelerator’s DRAM
via DMA and the AI cores load and store data from the DRAM as well. Thus, to
get complete timing estimates we use DRAMSim3 [78, 79], a state-of-the-art
cycle-accurate DRAM simulator. We model the DDR4-3200-x8 configuration and
generate memory traces (example [80]) of the DRAM subsystem. We combine DRAM
traces with the functional execution traces from the cores, and use the event-
based simulation to get end-to-end timing.

###  7.2 Adding TEE Support: SSD

Simulating the base model. The base model of the SSD simulation is based on
SimpleSSD [81, 82], a state-of-the-art cycle-accurate full-system simulator
that closely simulates commercial SATA and NVMe SSDs with realistic
performance characteristics.

Hardware changes. Our design does not necessitate changes to the SSD
controller. We implement ACU, and FMT in $259$ LoC of C++ directly to
SimpleSSD’s codebase ($\sim 55K$). This allows us to perform functional
simulation based on Gem5 [83] that boots Linux and performs IO operations with
the SSD. We implement these in $150$ LoC of SystemVerilog RTL synthesized and
run place-and-route on 28nm and 150 LoC in python to verify the RTL using
cocotb. We use this implementation to estimate the timing costs per operation
and we update the SimpleSSD’s timing model accordingly. We also update the
timing model with the costs for cryptographic operations performed by the MPE.
We use the enhanced timing model along with the functional simulation to get
accurate execution-timing traces which we use to estimate the runtime
overhead.

###  7.3 Security Controller (SC)

We implement a software SC prototype in C++. We use libsodium for AES256-GCM
with Intel’s AES-NI C intrinsic to accelerate AES, and Intel’s AVX2 for fast
memory copy (_mm256_stream_si256). Our prototype implementation is single-
threaded and handles a limited number of devices as the main bottleneck is
AES-GCM bandwidth. It is straightforward to extend it to multiple cores of
multi-channel memory to scale the performance with an increasing number of
connected nodes. We add around $400$ LoC of wrapper code along with unmodified
libsodium library and a runtime for communication between TEE and non-TEE
nodes

##  8 Evaluation

Table 1: Evaluation Platforms

Evaluation Hardware OS/RT/Driver AI acc & DRAM sim 2$\times$Intel Xeon Gold
6240R, 1 TB DRAM Ubuntu 20.04.5 LTS AI acc Huawei Atlas 300T (Ascend 910 SoC,
32 core) Ascend runtime SSD sim & SC Intel i9 11900K, 128 GB DRAM Ubuntu
20.04.5 LTS GPU NVIDIA GeForce RTX 3080 Nvidia driver GPU & FPGA host Intel
Xeon $2^{nd}$ Gen Scalable CPU Ubuntu 20.04 LTS FPGA ShEF AWS F1 instances AWS
+ Xilinx XDMA FPGA local Xilinx UltraScale+ VCU11 Xilinx XDMA

Our Setup. Table 1 describes the hardware platforms, OS, runtime, driver we
use. From an experimental standpoint, the amount of time it took us to run AI
accelerator simulation was high. Specifically, an execution that requires $1$
second in real-clock time, takes us $\sim 20$K seconds ( $\sim 5.5$ hours) to
simulate. Further, this rate increases linearly with the input data size. In
comparison to the AI accelerator, the simulation for SSD is faster—$1$ second
in real-clock time takes us $\sim 24$ seconds to simulate. Booting up Linux
takes $\sim 25$ minutes on x86 and $\sim 60$ minutes on ARM. For evaluation,
we select benchmarks and workloads that reflect the most popular real-world
computations deployed on cloud accelerators.

###  8.1 Case Study 1: AI Accelerator

Synthetic stress test benchmarks. We develop two synthetic workloads: i) one
that is memory-bound, ii) and one that is compute-bound. To build these
workloads, we use Im2col [84], the dominant computation of the core
responsible for CNN layers [85]. Table 2 shows the number of cycles required
for both memory and compute-bound applications. We see that the memory-bound
workload incurs significantly more overhead on the baseline than the compute-
bound workload. We observe that the overhead incurred on memory-bound
computation with protected access (P acc) is $\sim 9\times$ higher than the
corresponding compute-bound workload. This is expected as such memory bound
computation engages the protection units (ACU and MPE) more than compute-bound
applications. The SDA unit’s performance depends on the available memory
bandwidth of a core and the memory subsystem. The two computations use
similarly sized tensors. Therefore, the latency of SDA is similar for both
memory (1901 cycles) and compute-bound (2125 cycles) workloads.

Real-world AI workloads experiment setup. Table 3 provides the results for
loading the models into the accelerator, running the image inference on
CIFAR-10($D_{1}$) [86] ($60k$ $32\times 32$ RGB 163MB), and ImageNet
ILSVRC($D_{2}$) [87] (1.4M $224\times 224$ RGB, 157GB) .

Table 2: Effect of protected access ACU+MPE (P acc) and secure de-allocation
(SDA) on AI Accelerator. NP: non-protected access.

Memory bound (Filter dim = 1)  Compute bound (Filter dim = 3) Access type
Cycles Overhead  Access type Cycles Overhead NP acc 5821 Baseline  NP acc
13552 Baseline NP acc + SDA 7722 32.65%  NP acc + SDA 15677 15.68% P acc 6688
14.89%  P acc 13782 1.7% P acc + SDA 8847 51.98%  P acc + SDA 16198 19.54%

Setup and Teardown. We measure the time to transfer the AES256-GCM-protected
compiled models and instantiate the FMT with access control information.
Decryption & tag verification incur an average overhead of 12.9% as compared
to the baseline without these protections. The deallocator tracks all tensors
during execution and only clears up tensor memory during teardown. Therefore,
the teardown costs increase with the size of the model. However, these costs
are independent of the data set size because the input data are well as the
results are written directly into the tensor memory E.g., SSD-VGG-16, a large
model(380.2 MB), and Resnet34, a smaller model (105MB) have teardown costs of
8.08 ms and 2.23 ms respectively.

Table 3: AI Accelerator case study. The table shows the effect of adding TEE
support in the Ascend AI accelerator.
$L_{act}$: #activation and model parameter $D_{1}$: DMA time of batch size
$\times$ CIFAR-10 image(s) Setup: DMA time of model (%overhead) Base:
Unmodified accelerator $L_{bin}$: #model binary (.om file) $D_{2}$: DMA time
of batch size $\times$ ImageNet image(s) Teardown: SDA time after enclave exit
Secure: Base+protection units

Model Information | One time cost (ms) | Runtime costs (On a single DaVinci core)
---|---|---
Static Model Information | Compiled (MB) | Set up | Teardown | Base DMA($\mu$s) | Secure DMA ($\mu$s) | Inference Runtime (ms) | #Memory Access
Model | Batch | Res | # Layers | $L_{act}$ | $L_{bin}$ | Base | Secure(%) | Secure | $D_{1}$ | $D_{2}$ | $D_{1}$ | $D_{2}$ | CPU (64C) | Base | Secure | Overhead | Load | Store
Resnet-34 | $1$ | $224$ | $34$ | $2$ | $103$ | $8.2$ | 9.31 (13.5) | $2.23$ | $4.47$ | $106$ | $19$ | $458$ | $15.29$ | $1.206$ | $1.2437$ | 3.12% | $310\,382$ | $4846$
$8$ | $224$ | $34$ | $2$ | $103$ | $8.2$ | 9.31 (13.5) | $2.23$ | $35.76$ | $848$ | $152$ | $3664$ | $15.46$ | $6.0726$ | $6.13$ | 0.09% | $2\,482\,146$ | $39\,202$
$16$ | $224$ | $34$ | $2$ | $103$ | $8.2$ | 9.31 (13.5) | $2.23$ | $71.52$ | $1696$ | $304$ | $7328$ | $15.52$ | $11.755$ | $11.823$ | 0.05% | $4\,964\,290$ | $78\,402$
Resnet-50 | $1$ | $224$ | $50$ | $3.5$ | $108$ | $8.71$ | 9.84 (12.9) | $2.37$ | $4.47$ | $106$ | $19$ | $458$ | $22.11$ | $1.836$ | $1.909$ | 3.97% | $310\,382$ | $4846$
$8$ | $224$ | $50$ | $3.5$ | $108$ | $8.71$ | 9.84 (12.9) | $2.37$ | $35.76$ | $848$ | $152$ | $3664$ | $23.13$ | $10.706$ | $10.87$ | 1.53% | $2\,482\,146$ | $39\,202$
$16$ | $224$ | $50$ | $3.5$ | $108$ | $8.71$ | 9.84 (12.9) | $2.37$ | $71.52$ | $1696$ | $304$ | $7328$ | $23.29$ | $24.195$ | $24.667$ | 1.95% | $4\,964\,290$ | $78\,402$
RetinaNet-RN50-fpn | $1$ | $800$ | $50$ | $3.5$ | $228.4$ | $18.11$ | 20.38 (12.5) | $4.93$ | $4.47$ | $106$ | $4.47$ | $106$ | $329.56$ | $30.515$ | $30.684$ | 0.55% | $4\,948\,128$ | $278\,228$
SSD300-VGG-16 | $1$ | $300$ | $32$ | $115.2$ | $265$ | $29.7$ | 33.47 (12.6) | $8.08$ | $4.47$ | $106$ | $4.47$ | $106$ | $37.39$ | $6.539$ | $6.631$ | 1.4% | $1\,199\,912$ | $75\,176$
UNet | $1$ | $572$ | $23$ | $8$ | $119$ | $9.92$ | 11.24 (13.3) | $2.7$ | $4.47$ | $106$ | $4.47$ | $106$ | $224.89$ | $23.379$ | $23.479$ | 0.42% | $3\,077\,286$ | $187\,302$
YOLO V3 | $1$ | $256$ | $106$ | $50$ | $237$ | $22.42$ | 25.30(12.86) | $6.1$ | $4.47$ | $106$ | $19$ | $458$ | $92.9$ | $3.606$ | $3.747$ | 3.91% | $150\,145$ | $9857$
$1$ | $416$ | $106$ | $50$ | $237$ | $22.42$ | 25.30(12.86) | $6.1$ | $4.47$ | $106$ | $19$ | $458$ | $1386.19$ | $7.591$ | $7.709$ | 1.55% | $396\,475$ | $26\,027$
$8$ | $256$ | $106$ | $50$ | $237$ | $22.42$ | 25.30(12.86) | $6.1$ | $35.76$ | $848$ | $152$ | $3664$ | $3400.99$ | $19.23$ | $19.45$ | 1.44% | $1\,201\,153$ | $78\,849$

Runtime data copy costs. To understand the impact of securing the DMA
transfers between the host and the AI accelerator we measure the time taken to
load state-of-the-art AI networks to the accelerator memory with and without
AES256-GCM protection performed by the MPE. Table 3 shows the transfer time
and overhead for the two datasets. We evaluate all models with the same 2
datasets. So, the average DMA transfer time for a given a data set and batch
size is the same across models.

For a 4KB DMA transfer (after cache warm-up), AES256-GCM takes $\sim$1.47
$\mu$s on one Intel i9 11900K CPU core at 5.2 GHz, and $\sim$3.52 $\mu$s on
one ARM v8 Cortex-A CPU core at 800 MHz. Therefore, we incur noticeable
overheads for runtime data transfer that employs a large number of decryption
operations. However, a single round of inference takes more time. E.g., For
Resnet-34 and batch size = 16, ImageNet transfer takes $7.32$ ms and the
inference takes $11.82$ ms. In these cases, we pipeline the data transfer and
inference (initiating the next transfer while the previous is still in
execution on the accelerator ) such that total execution time is only $11.82$
ms.

Table 4: Multi-tenant scaling. Micro-benchmark of load/store latency in cycles
(%slowdown) from AI cores to DDR. Sequential (S)/random (R) Read & Write (4K)
bandwidth (MB/s/tenant) (%scaling) for Samsung Z-SSD.

AI accelerator SSD #Tenants 1 2 4 1 2 4 Load 294 307.1 (4.4) 316.5 (7.4) S/R
Read 383/372 191(50)/186(50) 92(75)/93(75) Store 255.5 257.7(0.7) 316.5(23.4)
S/R Write 410/410 202(49)/205(50) 97(75)/97(75)

Results. To enable protected access (ACU+MPE) we see a worst-case overhead of
$14.89$% for the synthetic memory-bound workloads (P acc), but only 3.98% for
a real-world AI workload (Resnet-50, batch size = 1). This shows that real-
world workloads do not exhibit worst-case performance as they are neither
purely memory or compute bound. Additionally, real-world workloads maximize
data reuse by caching, which reduces the number of load and store instructions
issued by the cores. This is apparent in all the scenarios (CPU, Base
accelerator and Secure i.e., Base+ACU) in Table 3 where the average running
time for a single image decreases with increasing batch size. E.g., in
Resnet34 runtime overhead decreases from 3.12% (batch=1) to 0.05% (batch=16).

Resnet-34 and Resnet-50 have the same number of load and store operation
despite different inference time as they have the exact same memory access
pattern. However, Resnet-50 runs for more iterations, and in these iterations
accesses cached data. This results in an increase in runtime but does not
increase the number of load and store operations. For teardown, we observe a
very small average overhead for $1.7\%$ when running inference on the Imagenet
dataset. Additionally, we observe that running the inference on a CPU with
$64$ logical cores is always slower on average by a factor of $37.66\times$
than running it on the just $1$ core of the AI accelerator.

Scalability. We run real-world workloads with increasing number of tenants. As
the tenants are pinned to specific AI cores until the enclaves exit, there are
no context switch overheads. Therefore, the bandwidth between the core and DDR
could only be the bottleneck factor. This bandwidth can be saturated if all
the tenants issue load and store simultaneously. We see that the real-world
workloads with $4$ tenants does not saturate the large memory bandwidth ($47$
GBps). Therefore, we do not observe any noticeable performance degradation for
real-world AI workloads with $4$ number of simultaneous tenants. To analyze
the worst-case scenario, we run the synthetic memory-bound stress test for
1,2, and 4 concurrent tenants as shown in table Table 4. When the AI core does
not perform any meaningful computation but continuously issues load and store
operations, we see a slowdown of $\sim 30\%$ with $4$ tenants.

Figure 6: Floor plan of the synthesized hardware protection module on the AI
accelerator alongside power [88] and area breakdown.

Area overhead. Fig. 6 shows the synthesized, placed & routed floor plan of the
hardware design that implements isolation mechanisms in the memory and the
interface. The hardware only consumes 0.38 $\mu$W ($1.15\times 10^{-7}\%$)
power and 1228 $mm^{2}$ ($0.2\%$) die area as compared to Ascend 910 which
consumes 300W power and has a die area of 2786 $\mu m^{2}$.

Table 5: SSD case study. Block IO (Read/Write) latency. ISO - FDU isolation,
ENC - AES256-GCM on six benchmarks: sequential reads (S-R), sequential writes
(R-W), random reads (R-R), read-writes (R-W), sequential read-writes (S-RW),
and random read-writes (R-RW)

S-R S-W R-R R-W S-RW R-RW SSD OP cost Base $\mu$s ISO % ENC % Base $\mu$s ISO
% ENC % Base $\mu$s ISO % ENC % Base $\mu$s ISO % ENC % Base $\mu$s ISO % ENC
% Base $\mu$s ISO % ENC % Intel 700 400 G ISO=5ns ENC=10.24 $\mu$s $13.39$
$0.074$ $76.42$ $9.35$ $0.1$ $109.4$ $110.2$ $0.009$ $9.29$ $9.35$ $0.1$
$109.4$ $58.26$ $0.017$ $17.57$ $120.36$ $0.008$ $8.5$ NVMe Samsung Z-SSD 400G
ISO=5ns ENC=10.24 $\mu$s $4.4$ $0.113$ $115.89$ $6.22$ $0.080$ $82.3$ $13.28$
$0.037$ $38.54$ $6.22$ $0.080$ $82.3$ $10.14$ $0.049$ $50.45$ $11.55$ $0.043$
$44.32$ Samsung 983DCT 1.92T ISO=5ns ENC=10.24 $\mu$s $12.39$ $0.04$ $41.29$
$8.7$ $0.056$ $58.27$ $27.74$ $0.018$ $18.45$ $8.78$ $0.056$ $58.27$ $17.78$
$0.028$ $28.79$ $83.17$ $0.006$ $6.155$ SATA Samsung 850Pro 256G ISO=10ns
ENC=10.24 $\mu$s $83.17$ $0.006$ $6.15$ $415.9$ $0.002$ $2.46$ $412.87$
$0.002$ $2.48$ $419.72$ $0.009$ $9.29$ $419.56$ $0.023$ $2.44$ $416.75$
$0.0023$ $2.45$ Intel 535 240G ISO=13.33ns ENC=10.24 $\mu$s $560.1$ $0.0023$
$2.43$ $1875$ $0.0007$ $0.72$ $727.4$ $0.002$ $2.43$ $2063$ $0.0006$ $0.66$
$1017$ $0.001$ $1.34$ $1184$ $0.001$ $1.15$

###  8.2 Case Study 2: SSD

SSD Models & Workloads. We evaluate our design on $5$ types of SSD models
provided in SimpleSSD. We tested them with workloads from FIO, a standard tool
for storage benchmark [89] that perform six distinct disk operations ( Table
5) on 2-16 GB of data. In FIO, we use the default 4KB block size (complies
with Advanced Format standard [90]). We perform a warmup for DRAM cache before
our measurements, to reflect real-world workloads. We do not use any file
system to avoid noises created by optimizations, however, we configure FIO to
imitate LinuxFS behavior with an IO depth of 32.

Performance Overheads. The performance of the SSD only depends on the block
size, as it processes them one at a time in a streaming fashion. Thus, the
total size of data transferred does not affect SSD’s performance. Therefore,
we show the average latency overhead for 4K blocks in Table 5. Given an SSD,
we observe that all operations incur a fixed isolation and encryption
overhead. Further, as expected this latency is inversely proportional to the
frequency of the ARM core on the SSD. A detailed specification of the SSDs
used in our evaluation is shown in Sec. A.6. In the baseline, sequential read
and sequential read-write are faster for than their random counterparts (e.g.,
on Intel 700, S-R takes 13.39 $\mu$s compared to $110.2\mu$s for R-R) as read
operation take advantage of the caches. In contrast, write operations do not
have cache advantages as the caches have a write-through policy to guarantee
data persistence on the SSD. Therefore, the sequential and random writes have
similar latency. The MPE cannot use the caches as it performs AES-GCM before
the data is stored in the cache. Hence, the overheads for AES-GCM as
considerably high with the highest average of 31.93% for S-RW.

Scalability. We instantiate 1, 2, and 4 simultaneous FIO [89] benchmarks on
the SSD and measure the read-write bandwidth. We observe that adding ACU and
MPE does not significantly reduce the bandwidth. E.g., the baseline bandwidth
for sequential read for Samsung Z-SSD 800G SSD is $383.9$ MB/s and adding our
protection does not change it as shown in table 4.

Area overhead. Compared to AI accelerator, the area overhead is significantly
smaller due to simpler design. The hardware only take 650 $\mu m^{2}$ on the
SSD controller.

###  8.3 Case Study 3 & 4: GPUs and FPGA

GPU MMIO and DMA. our design requires hooking on MMIO transfers (refer to Sec.
4.2) to protect them using AES-GCM. We use the Imagenet implementation and
TinyImagenet dataset from the PyTorch library and perform training of
different models as shown in Table 6. To understand when MMIO is used in
different GPUs and to estimate the overhead for protecting them we use the
mmiotrace utility in the Linux kernel to log all MMIO between application
memory and GPU. We count the number of MMIO reads and writes to the driver-
allocated application-specific memory-mapped region. We observe that MMIO is
used to load model and data and there is no MMIO during the training and
validation cycles. Specifically, $38\%$, $42\%$ and $20\%$ for setup, load
data& model, and teardown. Therefore, protecting these MMIO accesses will not
incur any performance costs for the actual computation. Further, we use the
traces to estimate $1.89\%$ total overhead of adding AES256-GCM.

To capture DMA requests to and from the GPU, we use Nvidia’s profiling utility
nsys. We see that on the CPU, cudaMemCopy commands are issued before and after
the kernels are executed. To protect DMA we would require the driver to hook
on these calls, encrypt the data, and copy it to non-secure memory. Similar to
MMIO, this can be done before the kernels are launched. On a closer look, we
see from the events captured on the GPU that the data copies are performed in
parallel with the computation. We can perform streaming encryption to maintain
the speedups due to such streaming.

Table 6: GPU case study: The overhead% on MIMO accesses shows the effect of
adding TEE properties on the training process.

Model | Batch | Setup | LoadModel | LoadData | Teardown | Overhead%
---|---|---|---|---|---|---
Resnet-32 | 32 | 106763 | 65736 | 57804 | 56398 | 2.28
16 | 101683 | 64220 | 58435 | 57343 | 1.86
8 | 109822 | 63689 | 57212 | 56272 | 1.54
Resnet-50 | 32 | 114591 | 64970 | 57939 | 56523 | 2.12
16 | 104517 | 63233 | 57503 | 56365 | 1.82
8 | 108054 | 64596 | 56973 | 56011 | 1.26
Vgg16 | 32 | 106536 | 61420 | 57182 | 55824 | 1.18
Squeezenet1_1 | 32 | 111017 | 50923 | 56939 | 55623 | 2.54
Alexnet | 32 | 111971 | 64149 | 58329 | 57031 | 2.42

FPGA MMIO and DMA. To analyze the costs of MMIO and DMA protection for FPGAs,
we use DNNWeaver [91]. We execute on AWS F1 [92] instances with and without
the TEE feature provide by SheF [29]. We use mmiotrace and Xilinx DMA (xdma)
driver to capture all MMIO and DMA accesses respectively. We observe the same
data transfers sizes via DMA (1.41MB) and MMIO (16.91KB), with and without TEE
features enabled. The computation performs small-sized MMIOs ($4$B) to
initiate DMAs of size $4$KB to transfer models and data. During the FPGA
execution, we observe no MMIO accesses. We see a average of $4344$ MMIOs and
$362$ DMAs. To protect the transfers, we estimate similar overheads:
$0.0033\%$ for MMIO and $0.0005$% for DMA, with and without TEE features. We
also observe that on average, the time between consecutive MMIO and DMA
transfers are 10-20 $\mu$, 130 $\mu$s respectively. From our experiments, we
know that the time to compute AES256-GCM with a block size of 4KB is only 1.47
$\mu$s as measured in Sec. 8.1. Therefore, encryption will not become a
bottleneck for the MMIO and DMA transfers.

We implement SVD [93] using Xilinx’s Vitis libraries and run it on a local
Xilinx vcu118 FPGA. The application uses xdma to transfer data to and from the
FPGA. The FPGA computes the SVD of matrices ranging from $10\times 10$ to
$20000\times 20000$, transfers data from $1.6$KB to $6.4$GB, causes $52$ and
$9958$ MMIO,and $4$ to $766$ DMA operations respectively. Therefore, We
observe that the number of MMIO and DMA are linearly proportional to the size
of data. We use the MMIO and DMA traces to estimate the average latency
introduced for protecting these accesses to be $0.04\%$ for MMIO and $0.002\%$
for DMA.

###  8.4 Security Controller (SC)

To evaluate the scaling limit of SC, we consider SC’s individual operations.
First, the SC performs AES256-GCM of data for the non-TEE nodes. The measured
bandwidth of the native Intel AES-NI (using libsodium) AES256-GCM
implementation is 2.59 GB/s/core. Next, we measure the bandwidth for copying
data between encrypted and plaintext buffers (using Intel AVX2) to be 25
GB/s/core. Resnet-34, the fastest inference network that we evaluate, takes
1.2 ms/image/AI core. With the average size of 170KB/image in the ImageNet
dataset, an AI core can be saturated with a image data rate of 138 MB/s from
the CPU FDU. Using this data, we estimate that the SC with a buffer size of
2.5 GB, on a single core can handle 19 CPU-FDUs concurrently streaming image
data. Adding jobs requires transferring the compiled trained model to the
accelerator and is expensive, but it’s a one-time cost per job. With the
largest model that we evaluate on (SSD300-VGG-16 which is 380 MB), SC on a
single core can handle 6.97 jobs/sec without performance degradation.

The number of nodes that can be connected to the SC and the buffer sizes
required to support them, scales linearly with the number of CPU cores the SC
uses. E.g., instead of a single core, we can deploy the SC on an Intel Xeon
Gold processor with 48 cores as long as the memory system has enough bandwidth
to support the SC buffers (171.5 GB/s for DDR4 8-channel 3200MHz). Such a
multi-core SC deployment theoretically can handle up to 912 nodes or 334
nodes/sec simultaneously for inference or starting up jobs respectively.

##  9 Related Work

Building a scalable infrastructure for multi-tenant cloud has been a
challenging distributed systems problem. Prior works have shown solutions [94,
95] that are now deployed in production. However, they assume a trusted cloud
service provide and only protected against a software attacker executing as a
malicious tenant. Our work focuses on stronger adversary model: untrusted
cloud provider and a physical attacker.

Device TEEs. Existing proposals enable TEE on devices such as GPUs [26, 96,
97, 98, 99], FPGAs [29, 30, 100], SSDs [28], and other accelerators [101].
They focus solely on adding TEE-features so as to create hardware-enabled
enclaves on the accelerator—either for multi-tenancy or for single-tenancy.
They assume that the device is connected to a single physical host machine
that may optionally have TEE support. They primarily achieve isolation from
the attacker-controlled driver, OS, and hypervisor. Some device TEE solutions
additionally remove the device driver from the TCB and provide protection
against a physical adversary. The only known commercially announced non-CPU
TEE is NVIDIA’s upcoming H100 [27] which enables secure isolated instances
(known as MIG [50]) in the GPU using proprietary GPU firmware. Additionally,
prior works use cryptographic primitives like homomorphic encryption [102],
build privacy-preserving machine-learning frameworks [103, 104], and oblivious
multi-party computation for data-centers [105].

Heterogeneous TEE. Prior proposals enable TEEs across a CPU-TEE and a device-
TEE physically connected to each other. SGX-FPGA [106] proposes a custom TEE
for FPGA. It uses a physical unclonable function (PUF) on the FPGA for
hardware root of trust. The FPGA is managed by a trusted SM that communicates
with the SGX enclave over an encrypted channel. HIX [97] achieves similar
protection for GPUs where an SGX enclave communicates with the GPU enclave
over encrypted shared memory using HIX trusted driver. Hector-V [107] provides
a switch wrapper configured during boot time by the CPU-SM. It acts as a
filter for peripherals on the AXI4-lite bus to drop request coming from
untrusted code. Another line of research has shown the feasibility of doing
enclaved execution on unmodified devices by extending the CPU-TEE protection.
They leverage MMU protection to protect memory-mapped devices and perform bus-
level isolation since devices are physically connected to a TEE host [35, 36,
37].

Our solution is compatible with these device-TEEs, but does not require the
devices to be directly connected to CPU-TEE hosts. We use the insights from
above proposals to build the TEE-support for AI accelerator and SSD, by adding
protection units (ACU, FMT, MPE) and relying on security monitors. Our SC-
based access control for distributed setting over a network mimics the bus-
level filtering for a centralized setting where CPU and devices are connected
a bus.

Distributed TEE. HETEE [38] addresses a similar problem as ours, but in a
different setting. It assumes non-TEE devices that makes it more general.
However, it also assumes that devices are connected to a centralized security
controller (SC) over a PCIe switch in a physical attack-proof container. This
limits the scalability of HETEE to a single rack and $\sim 60$ nodes. In
contrast, our design carefully avoids a centralized SC design ($S_{0}$ in Fig.
2 in Sec. 3.1). Nonetheless, at least at the rack-level, our solution may look
similar to HETEE. HETEE’s centralized SC solution leads to bottlenecks even
within the rack. First, it does not allow multi-tenancy on any nodes, since
they do not have TEE support. We allow multi-tenancy on TEE-nodes. Second, the
SC has to perform encryption decryption operations for all data between 60
nodes. Hence, they employ trusted proxy nodes to evenly distribute the
cryptographic operations. We avoid this problem by protecting the non-TEE
nodes by the SC’s access-control checks, encryption decryption is only
required when communicating with TEE nodes.

Commercial confidential cloud solutions. Several CSPs offer accelerators as a
unit [92, 108] or as a packaged service [109, 110]. However, none of them are
TEE-protected as of yet. CPU and hypervisor-based TEEs have become prominent.
E.g., SGX & SEV-based Microsoft Azure confidential cloud [31], hypervisor-
based Amazon Nitro [34], SEV-based Google confidential GTK [32], and SGX-based
IBM confidential cloud [33]. Azure announced confidential computing on NVIDIA
A100s GPUs that allows isolated computation on a dedicated GPU connected to a
AMD SEV protected CPU-TEE.

##  10 Conclusion

We present a data-center design that provides enclaved execution guarantees
over a mix of TEE and non-TEE devices. Our hybrid design pays off by scaling
across several nodes as well as being future-ready for TEE-enabled devices.
Our evaluation shows that the performance costs of such a design are modest
and can be reduced further with fine-tuning. We envision that our insight will
serve as guiding principles for cloud-scale solutions for confidential
computing beyond CPU execution.

## References

  * [1] K. Katrinis, D. Syrivelis, D. Pnevmatikatos, G. Zervas, D. Theodoropoulos, I. Koutsopoulos, K. Hasharoni, D. Raho, C. Pinto, F. Espina, S. Lopez-Buedo, Q. Chen, M. Nemirovsky, D. Roca, H. Klos, and T. Berends, “Rack-scale disaggregated cloud data centers: The dredbox project vision,” in 2016 Design, Automation Test in Europe Conference Exhibition (DATE), pp. 690–695, 2016.
  * [2] NVIDIA, “Nvidia A100 gpus power the modern data center.” https://www.nvidia.com/en-us/data-center/a100/.
  * [3] S. Yesil, M. M. Ozdal, T. Kim, A. Ayupov, S. Burns, and O. Ozturk, “Hardware accelerator design for data centers,” in 2015 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 770–775, IEEE, 2015.
  * [4] T. J. Ham, L. Wu, N. Sundaram, N. Satish, and M. Martonosi, “Graphicionado: A high-performance and energy-efficient accelerator for graph analytics,” in 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1–13, IEEE, 2016.
  * [5] run.ai, “Keras multi gpu: A practical guide.” https://www.run.ai/guides/multi-gpu/keras-multi-gpu-a-practical-guide.
  * [6] S. Choi, S. Lee, Y. Kim, J. Park, Y. Kwon, and J. Huh, “Serving heterogeneous machine learning models on Multi-GPU servers with Spatio-Temporal sharing,” in 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 199–216, 2022.
  * [7] X. Zhu, W. Chen, W. Zheng, and X. Ma, “Gemini: A Computation-Centric distributed graph processing system,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 301–316, 2016.
  * [8] J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. Franklin, and I. Stoica, “GraphX: Graph processing in a distributed dataflow framework,” in 11th USENIX symposium on operating systems design and implementation (OSDI 14), pp. 599–613, 2014.
  * [9] M. Lavasani, H. Angepat, and D. Chiou, “An fpga-based in-line accelerator for memcached,” IEEE Computer Architecture Letters, vol. 13, no. 2, pp. 57–60, 2013.
  * [10] S. R. Chalamalasetti, K. Lim, M. Wright, A. AuYoung, P. Ranganathan, and M. Margala, “An fpga memcached appliance,” in Proceedings of the ACM/SIGDA international symposium on Field programmable gate arrays, pp. 245–254, 2013.
  * [11] H. Eran, L. Zeno, M. Tork, G. Malka, and M. Silberstein, “NICA: An infrastructure for inline acceleration of network applications,” in 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 345–362, 2019.
  * [12] Y. Lu, G. Chen, B. Li, K. Tan, Y. Xiong, P. Cheng, J. Zhang, E. Chen, and T. Moscibroda, “Multi-Path transport for RDMA in datacenters,” in 15th USENIX symposium on networked systems design and implementation (NSDI 18), pp. 357–371, 2018.
  * [13] A. Singhvi, A. Akella, D. Gibson, T. F. Wenisch, M. Wong-Chan, S. Clark, M. M. Martin, M. McLaren, P. Chandra, R. Cauble, et al., “1rma: Re-envisioning remote memory access for multi-tenant datacenters,” in Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, pp. 708–721, 2020.
  * [14] H. AlJahdali, A. Albatli, P. Garraghan, P. Townend, L. Lau, and J. Xu, “Multi-tenancy in cloud computing,” in 2014 IEEE 8th international symposium on service oriented system engineering, pp. 344–351, IEEE, 2014.
  * [15] M. Adler, K. E. Fleming, A. Parashar, M. Pellauer, and J. Emer, “Leap scratchpads: automatic memory and cache management for reconfigurable logic,” in Proceedings of the 19th ACM/SIGDA international symposium on Field programmable gate arrays, pp. 25–28, 2011.
  * [16] V. Costan and S. Devadas, “Intel sgx explained.,” IACR Cryptology ePrint Archive, vol. 2016, no. 086, pp. 1–118, 2016.
  * [17] J. Winter, “Trusted computing building blocks for embedded linux-based arm trustzone platforms,” in Proceedings of the 3rd ACM workshop on Scalable trusted computing, pp. 21–30, 2008.
  * [18] V. Costan, I. Lebedev, and S. Devadas, “Sanctum: Minimal hardware extensions for strong software isolation,” in 25th USENIX Security Symposium (USENIX Security 16), pp. 857–874, 2016.
  * [19] K. Suzaki, K. Iijima, T. Yagi, and C. Artho, “Memory deduplication as a threat to the guest os,” in Proceedings of the Fourth European Workshop on System Security, pp. 1–6, 2011.
  * [20] Intel, “Intel software guard extensions.” https://software.intel.com/content/www/us/en/develop/topics/software-guard-extensions.html.
  * [21] A. Ltd., “Learn the architecture: Trustzone for aarch64.” https://developer.arm.com/architectures/learn-the-architecture/trustzone-for-aarch64/trustzone-in-the-processor, 2021\.
  * [22] D. Lee, D. Kohlbrenner, S. Shinde, K. Asanović, and D. Song, “Keystone: An open framework for architecting trusted execution environments,” in Proceedings of the Fifteenth European Conference on Computer Systems, pp. 1–16, 2020.
  * [23] Intel, “Intel trust domain extensions (intel tdx).” https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html.
  * [24] AMD, “AMD SEV-SNP.” https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf.
  * [25] ARM, “Arm confidential compute architecture (arm-cca).” https://www.arm.com/why-arm/architecture/security-features/arm-confidential-compute-architecture.
  * [26] S. Volos, K. Vaswani, and R. Bruno, “Graviton: Trusted execution environments on gpus,” in 13th USENIX Symposium on Operating Systems Design and Implementation (osdi 18), pp. 681–696, 2018.
  * [27] M. Andersch, G. Palmer, R. K. N. Stam, V. M. G. Brito, and S. Ramaswamy, “NVIDIA Hopper Architecture In-Depth.” https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/, March 2022.
  * [28] L. Kang, Y. Xue, W. Jia, X. Wang, J. Kim, C. Youn, M. J. Kang, H. J. Lim, B. Jacob, and J. Huang, “Iceclave: A trusted execution environment for in-storage computing,” in MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 199–211, 2021.
  * [29] M. Zhao, M. Gao, and C. Kozyrakis, “Shef: shielded enclaves for cloud fpgas,” in Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 1070–1085, 2022\.
  * [30] S. Zeitouni, J. Vliegen, T. Frassetto, D. Koch, A.-R. Sadeghi, and N. Mentens, “Trusted configuration in cloud fpgas,” in 2021 IEEE 29th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pp. 233–241, IEEE, 2021.
  * [31] Microsoft, “Azure confidential cloud - protect data in use | microsoft azure.” https://azure.microsoft.com/en-us/solutions/confidential-compute/.
  * [32] Google, “Confidential computing | google cloud.” https://cloud.google.com/confidential-computing.
  * [33] IBM, “Confidential computing for total privacy assurance – ibm.” https://www.ibm.com/cloud/smartpapers/confidential-computing-for-total-privacy-assurance/#protect-sensitive-data-in-use.
  * [34] Amazon, “Aws nitro enclaves - create additional isolation to further protect highly sensitive data within ec2 instances.” https://aws.amazon.com/ec2/nitro/nitro-enclaves/.
  * [35] M. Schneider, A. Dhar, I. Puddu, K. Kostiainen, and S. Čapkun, “Composite enclaves: Towards disaggregated trusted execution,” IACR Transactions on Cryptographic Hardware and Embedded Systems, pp. 630–656, 2022\.
  * [36] R. Bahmani, F. Brasser, G. Dessouky, P. Jauernig, M. Klimmek, A.-R. Sadeghi, and E. Stapf, “CURE: A security architecture with customizable and resilient enclaves,” in 30th USENIX Security Symposium (USENIX Security 21), 2021.
  * [37] J. Jiang, J. Qi, T. Shen, X. Chen, , S. Zhao, S. Wang, L. Chen, N. Zhang, X. Luo, and H. Cui, “Cronus: Fault-isolated, secure and high-performance heterogeneous computing for trusted execution environments,” in The 55th ACM/IEEE International Symposium on Microarchitecture,, 2022.
  * [38] J. Zhu, R. Hou, X. Wang, W. Wang, J. Cao, B. Zhao, Z. Wang, Y. Zhang, J. Ying, L. Zhang, et al., “Enabling rack-scale confidential computing using heterogeneous trusted execution environment,” in 2020 IEEE Symposium on Security and Privacy (SP), pp. 1450–1465, IEEE, 2020.
  * [39] N. Alachiotis, A. Andronikakis, O. Papadakis, D. Theodoropoulos, D. Pnevmatikatos, D. Syrivelis, A. Reale, K. Katrinis, G. Zervas, V. Mishra, et al., “dredbox: A disaggregated architectural perspective for data centers,” in Hardware Accelerators in Data Centers, pp. 35–56, Springer, 2019.
  * [40] J. Zhu, R. Hou, X. Wang, W. Wang, J. Cao, B. Zhao, Z. Wang, Y. Zhang, J. Ying, L. Zhang, et al., “Enabling rack-scale confidential computing using heterogeneous trusted execution environment,” in 2020 IEEE Symposium on Security and Privacy (SP), pp. 1450–1465, IEEE, 2020.
  * [41] S. Legtchenko, H. Williams, K. Razavi, A. Donnelly, R. Black, A. Douglas, N. Cheriere, D. Fryer, K. Mast, A. D. Brown, et al., “Understanding Rack-Scale disaggregated storage,” in 9th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 17), 2017.
  * [42] Fungible, “PCIe device disaggregation: Fungible, enabling hyperdisaggregation of compute and storage resources across data center scales.” https://www.fungible.com/technology-showcase/pcie-device-disaggregation/, 2021\.
  * [43] K. Lim, J. Chang, T. Mudge, P. Ranganathan, S. K. Reinhardt, and T. F. Wenisch, “Disaggregated memory for expansion and sharing in blade servers,” ACM SIGARCH computer architecture news, vol. 37, no. 3, pp. 267–278, 2009.
  * [44] R. Lin, Y. Cheng, M. De Andrade, L. Wosinska, and J. Chen, “Disaggregated data centers: Challenges and trade-offs,” IEEE Communications Magazine, vol. 58, no. 2, pp. 20–26, 2020.
  * [45] S.-Y. Tsai, Y. Shan, and Y. Zhang, “Disaggregating persistent memory and controlling them remotely: An exploration of passive disaggregated $\\{$Key-Value$\\}$ stores,” in 2020 USENIX Annual Technical Conference (USENIX ATC 20), pp. 33–48, 2020.
  * [46] Intel, “Pci express device security enhancements - pcie-device-security-enhancements.pdf.” https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf, 2018\.
  * [47] S. Checkoway and H. Shacham, “Iago attacks: why the system call api is a bad untrusted rpc interface,” ACM SIGARCH Computer Architecture News, vol. 41, no. 1, pp. 253–264, 2013.
  * [48] G. Chen and Y. Zhang, “MAGE: Mutual attestation for a group of enclaves without trusted third parties,” in 31st USENIX Security Symposium (USENIX Security 22), (Boston, MA), pp. 4095–4110, USENIX Association, Aug. 2022\.
  * [49] G. Chen, Y. Zhang, and T.-H. Lai, “Opera: Open remote attestation for intel’s secure enclaves,” in Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS ’19, (New York, NY, USA), p. 2317–2331, Association for Computing Machinery, 2019.
  * [50] Nvidia, “Nvidia multi-instance gpu user guide :: Nvidia tesla documentation.” https://docs.nvidia.com/datacenter/tesla/mig-user-guide/.
  * [51] D. H. Woo, N. H. Seong, D. L. Lewis, and H.-H. S. Lee, “An optimized 3d-stacked memory architecture by exploiting excessive, high-density tsv bandwidth,” in HPCA-16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture, pp. 1–12, IEEE, 2010.
  * [52] AMD, “Amd instinct mi250x accelerator.” https://www.amd.com/en/products/server-accelerators/instinct-mi250x, 2022\.
  * [53] D. Foley, “Nvlink, pascal and stacked memory: Feeding the appetite for big data.” https://developer.nvidia.com/blog/nvlink-pascal-stacked-memory-feeding-appetite-big-data/, 2014\.
  * [54] E. Beyne, D. Milojevic, G. Van der Plas, and G. Beyer, “3d soc integration, beyond 2.5 d chiplets,” in 2021 IEEE International Electron Devices Meeting (IEDM), pp. 3–6, IEEE, 2021.
  * [55] S. Na, S. Lee, Y. Kim, J. Park, and J. Huh, “Common counters: Compressed encryption counters for secure gpu memory,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 1–13, IEEE.
  * [56] S. Yuan, A. W. B. Yudha, Y. Solihin, and H. Zhou, “Analyzing secure memory architecture for gpus,” in 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 59–69, IEEE, 2021\.
  * [57] S. Lee, J. Kim, S. Na, J. Park, and J. Huh, “Tnpu: Supporting trusted execution with tree-less integrity protection for neural processing unit,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 229–243, IEEE, 2022.
  * [58] W. Hua, M. Umar, Z. Zhang, and G. E. Suh, “Guardnn: secure accelerator architecture for privacy-preserving deep learning,” in Proceedings of the 59th ACM/IEEE Design Automation Conference, pp. 349–354, 2022.
  * [59] W. Hua, M. Umar, Z. Zhang, and G. E. Suh, “Mgx: Near-zero overhead memory protection for data-intensive accelerators,” in Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, (New York, NY, USA), p. 726–741, Association for Computing Machinery, 2022.
  * [60] B. Nisarga and E. Peeters, “System-level tamper protection using msp mcus,” Texas Instruments, 2016.
  * [61] Southco, “Rack level security.” https://files.southco.com/static/documents/brochures/201701-BR-Rack-Level-Security-EN.pdf, 2022\.
  * [62] Xilinx, “Xilinx smartssd computational storage drive.” https://www.xilinx.com/content/dam/xilinx/publications/product-briefs/xilinx-smartssd-computational-storage-drive-product-brief.pdf, 2021\.
  * [63] S. Boboila, Y. Kim, S. S. Vazhkudai, P. Desnoyers, and G. M. Shipman, “Active flash: Out-of-core data analytics on flash storage,” in 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–12, IEEE, 2012\.
  * [64] J. Do, Y.-S. Kee, J. M. Patel, C. Park, K. Park, and D. J. DeWitt, “Query processing on smart ssds: Opportunities and challenges,” in Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp. 1221–1230, 2013.
  * [65] S. Kim, H. Oh, C. Park, S. Cho, S.-W. Lee, and B. Moon, “In-storage processing of database scans and joins,” Information Sciences, vol. 327, pp. 183–200, 2016.
  * [66] R. Kaplan, L. Yavits, and R. Ginosar, “Prins: Processing-in-storage acceleration of machine learning,” IEEE Transactions on Nanotechnology, vol. 17, no. 5, pp. 889–896, 2018.
  * [67] D. Park, J. Wang, and Y.-S. Kee, “In-storage computing for hadoop mapreduce framework: Challenges and possibilities,” IEEE Transactions on Computers, 2016.
  * [68] N. Systems, “Ngd systems.” https://ngdsystems.com/.
  * [69] Scaleflux, “Scaleflux.” https://scaleflux.com/.
  * [70] H. Liao, J. Tu, J. Xia, and X. Zhou, “Davinci: A scalable architecture for neural network computing.,” in Hot Chips Symposium, pp. 1–44, 2019.
  * [71] H. Liao, J. Tu, J. Xia, H. Liu, X. Zhou, H. Yuan, and Y. Hu, “Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 789–801, IEEE, 2021.
  * [72] Infinion, “Slb 9670vq2.0 - infinion technologies.” https://www.infineon.com/cms/en/product/security-smart-card-solutions/optiga-embedded-security-solutions/optiga-tpm/slb-9670vq2.0/, 2022\.
  * [73] J. Camenisch, L. Chen, M. Drijvers, A. Lehmann, D. Novick, and R. Urian, “One tpm to bind them all: Fixing tpm 2.0 for provably secure anonymous attestation,” in 2017 IEEE Symposium on Security and Privacy (SP), pp. 901–920, IEEE, 2017.
  * [74] R. Andri, B. Bussolino, A. Cipoletta, L. Cavigelli, and Z. Wang, “Going further with winograd convolutions: Tap-wise quantization for efficient inference on 4x4 tiles,” in Micro-22: 55th IEEE/ACM International Symposium on Microarchitecture (to appear), IEEE/ACM, 2022.  https://arxiv.org/abs/2209.12982.
  * [75] O. Villa, D. Lustig, Z. Yan, E. Bolotin, Y. Fu, N. Chatterjee, N. Jiang, and D. Nellans, “Need for speed: Experiences building a trustworthy system-level gpu simulator,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 868–880, IEEE, 2021.
  * [76] M. Nemani and F. Najm, “High-level area and power estimation for vlsi circuits,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 18, no. 6, pp. 697–713, 1999.
  * [77] P. D. Z. U. Dallas, “Design automation of analog vlsi lect9.” https://personal.utdallas.edu/~zhoud/EE%207v88/lect%209/Design%20Automation%20of%20Analog%20VLSI%20Lect9.pdf, 2022\.
  * [78] “umd-memsys/dramsim3: Dramsim3: a cycle-accurate, thermal-capable dram simulator.” https://github.com/umd-memsys/DRAMsim3, 2020.
  * [79] S. Li, Z. Yang, D. Reddy, A. Srivastava, and B. Jacob, “Dramsim3: A cycle-accurate, thermal-capable dram simulator,” IEEE Computer Architecture Letters, vol. 19, no. 2, pp. 106–109, 2020.
  * [80] umd memsys, “umd-memsys/dramsim3/example.trace – memory trace.” https://github.com/umd-memsys/DRAMsim3/blob/master/tests/example.trace, 2022\.
  * [81] CAMELab, “Simplessd - simplessd 2.0.12 documentation.” https://docs.simplessd.org/en/v2.0.12/index.html.
  * [82] M. Jung, J. Zhang, A. Abulila, M. Kwon, N. Shahidi, J. Shalf, N. S. Kim, and M. Kandemir, “Simplessd: Modeling solid state drives for holistic system simulation,” IEEE Computer Architecture Letters, vol. 17, no. 1, pp. 37–41, 2018.
  * [83] Gem5, “gem5: The gem5 simulator system.” https://www.gem5.org/, 2022.
  * [84] leonardoaraujosantos.gitbook.io, “Making faster - artificial intelligence.” https://leonardoaraujosantos.gitbook.io/artificial-inteligence/machine_learning/deep_learning/convolution_layer/making_faster, 2022\.
  * [85] H. Wang and C. Ma, “An optimization of im2col, an important method of cnns, based on continuous address access,” in 2021 IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE), pp. 314–320, IEEE, 2021.
  * [86] A. Krizhevsky et al., “Learning multiple layers of features from tiny images,” 2009.
  * [87] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [88] A. Burg, “Fundamentals of vlsi cmos power consumption.” https://ee222-winter18-01.courses.soe.ucsc.edu/system/files/attachments/EE222_W18_%20PowerConsumptionBasics.pdf, 2018\.
  * [89] J. Axboe, “fio - flexible i/o tester rev. 3.32.” https://fio.readthedocs.io/en/latest/fio_doc.html, 2017.
  * [90] IDEMA, “The advent of advanced format.” https://web.archive.org/web/20120510044435/http://www.idema.org/?page_id=2369, 2022\.
  * [91] DnnWeaver, “Dnnweaver v2.0: Open-source specialized computing stack for accelerating deep neural networks.” http://dnnweaver.org/, 2022.
  * [92] A. AWS, “Amazon ec2 f1 instances.” https://aws.amazon.com/ec2/instance-types/f1/, 2022.
  * [93] “Singular value decomposition (svd) tutorial.” https://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm, 2022\.
  * [94] Z. Shen, S. Subbiah, X. Gu, and J. Wilkes, “Cloudscale: elastic resource scaling for multi-tenant cloud systems,” in Proceedings of the 2nd ACM Symposium on Cloud Computing, pp. 1–14, 2011.
  * [95] R. Yu, G. Xue, V. T. Kilari, and X. Zhang, “Network function virtualization in the multi-tenant cloud,” IEEE Network, vol. 29, no. 3, pp. 42–47, 2015\.
  * [96] T. Hunt, Z. Jia, V. Miller, A. Szekely, Y. Hu, C. J. Rossbach, and E. Witchel, “Telekine: Secure computing with cloud $\\{$GPUs$\\}$,” in 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pp. 817–833, 2020.
  * [97] I. Jang, A. Tang, T. Kim, S. Sethumadhavan, and J. Huh, “Heterogeneous isolated execution for commodity gpus,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’19, (New York, NY, USA), p. 455–468, Association for Computing Machinery, 2019.
  * [98] L. K. Ng, S. S. Chow, A. P. Woo, D. P. Wong, and Y. Zhao, “Goten: Gpu-outsourcing trusted execution of neural network training,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14876–14883, 2021.
  * [99] O. Kwon, Y. Kim, J. Huh, and H. Yoon, “Zerokernel: Secure context-isolated execution on commodity gpus,” IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 4, pp. 1974–1988, 2019.
  * [100] H. Oh, K. Nam, S. Jeon, Y. Cho, and Y. Paek, “Meetgo: A trusted execution environment for remote applications on fpga,” IEEE Access, vol. 9, pp. 51313–51324, 2021.
  * [101] K. Vaswani, S. Volos, C. Fournet, A. N. Diaz, K. Gordon, B. Vembu, S. Webster, D. Chisnall, S. Kulkarni, G. Cunningham, et al., “Confidential machine learning within graphcore ipus,” arXiv preprint arXiv:2205.09005, 2022\.
  * [102] A. López-Alt, E. Tromer, and V. Vaikuntanathan, “On-the-fly multiparty computation on the cloud via multikey fully homomorphic encryption,” in Proceedings of the forty-fourth annual ACM symposium on Theory of computing, pp. 1219–1234, 2012.
  * [103] S. Tan, B. Knott, Y. Tian, and D. J. Wu, “Cryptgpu: Fast privacy-preserving machine learning on the gpu,” in 2021 IEEE Symposium on Security and Privacy (SP), pp. 1021–1038, IEEE, 2021.
  * [104] E. Hesamifard, H. Takabi, M. Ghasemi, and R. N. Wright, “Privacy-preserving machine learning as a service.,” Proc. Priv. Enhancing Technol., vol. 2018, no. 3, pp. 123–142, 2018.
  * [105] O. Ohrimenko, F. Schuster, C. Fournet, A. Mehta, S. Nowozin, K. Vaswani, and M. Costa, “Oblivious $\\{$Multi-Party$\\}$ machine learning on trusted processors,” in 25th USENIX Security Symposium (USENIX Security 16), pp. 619–636, 2016.
  * [106] K. Xia, Y. Luo, X. Xu, and S. Wei, “Sgx-fpga: Trusted execution environment for cpu-fpga heterogeneous architecture,” in 2021 58th ACM/IEEE Design Automation Conference (DAC), pp. 301–306, IEEE, 2021.
  * [107] P. Nasahl, R. Schilling, M. Werner, and S. Mangard, “Hector-v: A heterogeneous cpu architecture for a secure risc-v execution environment,” in Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, pp. 187–199, 2021.
  * [108] Microsoft, “Deploy ml models to field-programmable gate arrays (fpgas) with azure machine learning.” https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-deploy-fpga-web-service, 2022\.
  * [109] Huawei, “Modelarts - train ml models $\|$ huawei cloud.” https://www.huaweicloud.com/intl/en-us/product/modelarts.html, 2022.
  * [110] Google, “Ai and machine learning products $\|$ google cloud.” https://cloud.google.com/products/ai, 2022.
  * [111] A.-R. Sadeghi and C. Stüble, “Property-based attestation for computing platforms: caring about properties, not mechanisms,” in Proceedings of the 2004 workshop on New security paradigms, pp. 67–77, 2004.
  * [112] L. Chen, R. Landfermann, H. Löhr, M. Rohe, A.-R. Sadeghi, and C. Stüble, “A protocol for property-based attestation,” in Proceedings of the first ACM workshop on Scalable trusted computing, pp. 7–16, 2006.
  * [113] A. Seshadri, A. Perrig, L. Van Doorn, and P. Khosla, “Swatt: Software-based attestation for embedded devices,” in IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004, pp. 272–282, IEEE, 2004.
  * [114] Y. Li, J. M. McCune, and A. Perrig, “Sbap: Software-based attestation for peripherals,” in International Conference on Trust and Trustworthy Computing, pp. 16–29, Springer, 2010.
  * [115] A. Seshadri, M. Luk, and A. Perrig, “Sake: Software attestation for key establishment in sensor networks,” in International Conference on Distributed Computing in Sensor Systems, pp. 372–385, Springer, 2008.
  * [116] Y. Yang, X. Wang, S. Zhu, and G. Cao, “Distributed software-based attestation for node compromise detection in sensor networks,” in 2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007), pp. 219–230, IEEE, 2007.
  * [117] C. Castelluccia, A. Francillon, D. Perito, and C. Soriente, “On the difficulty of software-based attestation of embedded devices,” in Proceedings of the 16th ACM conference on Computer and communications security, pp. 400–409, 2009.

##  Appendix A Appendix

###  A.1 Enclave Manifest

⬇

{"Enclave" : "Enclave1",

"Enclave Vendor" : "Vendor1",

"Version" : "X.YZ",

"Resource" : {

{"Resource type : CPU", "Policies" : {"no-HT"}

"Cores" : 2, "Memory" : 256M },

{"Resource type : AI_Accelerator",

"Policies : {"memIsolation", "cachePartitioned"},

"Cores" : 20, "Memory" : 16G}

"SHA-3":"0xe3bae2603b...", "Sig":"18fd2fa796..."}

Figure 7: Example manifest file where the the programmer specifies the number,
size, and type of each FDU as well as the code to be executed on each FDU.

To launch an enclave with trusted FDUs and non-TEE nodes, the programmer uses
a CSP provided interface, to specify the number, size, and type of each FDU
and non-TEE node in a manifest file. Along with this, the programmer also
provides the code to be executed on each FDU and non-TEE node. One example
manifest is provide in Fig. 7. The manifest is signed by the private key of
the enclave developer to ensure that the integrity of the enclave manifest is
preserved. The remote verifier can use the manifest to verify the combined
attestation report (example in Fig. 11) to ensure that the allocations of the
FDUs and non-TEE node are correct.

###  A.2 DSA Configuration Attestation

One potential way to do the DSA attestation is to use software-based
attestation. However, a number of drawbacks in software-attestation (refer to
A.3) makes it not suitable in our case. The remote verifier starts the remote
attestation process by sending unique attestation challenges to each of the
local SMs. This process can be automated by a user-side software module that
can generate unique challenges. The attestation challenges ensure that the
attacker cannot reuse old attestation reports. First, the CPU SM generates the
attestation report of the enclave and public-private key pair for the enclave.
At the same time, the CPU SM forwards the DSA-specific challenges to the
allocated FDUs. Alongside traditional remote attestation, the DSAs leverage
_property-based attestation._ (PBA) [111, 112] that ensures if a DSA has
certain security properties. We opt for PBA as the DSAs are highly
programmable and attacker-controlled CSP or hypervisors can push malicious
configurations to the DSA to compromise user enclave data. Security properties
such as cache partitioning, disabled debug port etc. can enforced by the DSA-
SM. Therefore, the untrusted software stack cannot manipulate the device
configurations. During the measured boot process, the local SM updates the
platform configuration registers (PCR) with a pre-determined configuration
value that reflects the correct execution of a property as the following:

⬇

if(debug_port == false) //check for debug port status

TPM2_pcrExtend(23, 0xb2..);//extend 23rd PCR bank

else TPM2_pcrExtend(23, 0xf5..);

Figure 8: Committing predefined hash value to secure non-volatile memory on
the hardware security module (HSM) on DSA when certain security properties are
enforced in the firmware during startup.

ROOT: 0x235aa...Debug disabled: 0x566f..Core sep0xab20..Mem sep0x85ff..Debug
enabled : 0xd225f....Core sep0x5af3..Mem sep0x322a..PCR[23] ExtendRandom
challenge

Figure 9: Hierarchical PBA example with four leaf configurations that denotes
four states of a certain DSA’s firmware.

However, as the potential configurations can be extensive, we propose an
enhancement to the existing PBA scheme that we call _hierarchical-PBA_ that
executes during the DSA boot up phase. All FDUs on the same device can inherit
the device properties alongside FDU-specific configurations that can be
injected by the user before the enclave execution begins. Hierarchical-PBA
maintains one PCR bank that reflects a chain of properties that are initiated
from the user-generated attestation challenge. Figure 9 shows an example where
a DSA can have four different configuration represented by four leaf nodes.
Every configuration change (node in the tree) triggers a TPM2_pcrExtend
($PCR_{new}\leftarrow SHA(PCR_{old}\|value)$) that produces an unique PCR
value. The SM extends PCR bank every time there is a new security policy
enforcement on the code. The user can maintain a list of “whitelisted’
configuration values provided by the hardware vendor. Therefore, given a leaf
node value, the user can determine if a certain node has been initialized from
a proper state.

One concrete example of an attestation report from an AI accelerator is given
in Fig. 11, and corresponding security properties (part of the enclave
manifest) of the same accelerator is depicted in Fig. 10. This report is sent
back to the CPU-SM and then the CPU-SM sends the consolidated attestation
report back to the remote verifier.

⬇

{"device" : "Accelarator1", "manufacturer" : "Manuf1",

"firmwareVersion" : "X.YZ",

"deviceType" : "AI-Acc", "memoryRegion" : "256M"

"securityCapability" : {

"debug" : FALSE,

"coreIsolation" : TRUE,

"privateCache" : TRUE,

"sharedCache" : FALSE,

"sharedCacheIsolation" : FALSE,

"sharedMemoryIsolation" : TRUE,

"rowHammerMitigation" : FALSE

},

"Signature":"0x5005feb12c..."}

Figure 10: Example security properties of an accelerator, part of an enclave
manifest.

⬇

{"DeviceID" : "0x71f70e16a9ff9ca4ba14...",

"PublicKey" : "0xdcc026850848929d8b3...",

"Manufacturer" : "Manuf1",

"FirmwareVersion" : "X.YZ",

"DeviceType" : "type1",

# Accelerate properties from the manifest ((refer to the example in Fig. 10)%

"SecurityProperties" : {...},

# Platform configuration register values

"Configuration" : {

"PCR0" : "0xdcc0",

"PCR1" : "0xaeff",

...

"PCR23" : "0x1001"

}

"PolicyQuote" : "0xd0299944598e0f...",

# FDU properties)

"FDU" : {

"FDU_Support" : TRUE,

"Numbers" : 5,

"Config" : [

{"BAR" : "0xffff3225",

"streamID" : "0xc1fc8cd84",

"core_config" : 2,

"memory" : "128M"},

{...}

]

},

"FirmwareQuote":"0xdcc026850848929d8b3...",

"Certificate":"0x5005feb12ca9842eb60c8..."}

Figure 11: Example attestation result of an accelerator.

Key Distribution. As part of the attestation process, the user receives per-
FDU public keys that it can use to setup secure channels. Usually, along with
the run command to start the computation, user needs to send confidential data
for the computation as well. For this, it first generates per-FDU public-
private key pairs. Then, it uses the per-FDU public keys to generate a per-FDU
shared key (e.g., Diffie-Hellman ). This key is used to establish a secure
channel with every FDU. Further, the user has to provision a shared
communication key for all the FDUs. This communication key is used to send and
receive data and commands securely over the cloud infrastructure between the
FDUs. Finally, the user encrypts the data, run command, shared communication
key using the per-FDU shared secret. To this it adds the per-FDU public key it
generated and $\tt{manifestID}$ to the message and integrity protects it. It
then sends the message to the management plane. The management plane forwards
the message to the respective machines. The hypervisors on the individual
machines relay the message to the enclaves on the trusted FDUs. The enclave
first computes the shared secret using its own private key, provisioned by the
SM at startup and the public key of user in the message. Using this it
decrypts the data and the shared key for communication with other FDUs.

Revocation. We assume either the CSP or the user maintains a revocation list
of host, devices and their firmware versions. Such a list can be polled from
the CPU or device manufacturer signed by their root key.

###  A.3 Case for not using software-based attestation

In our proposal, we decided not to leverage any existing software-based
attestation [113, 114, 115, 116] for both TEE and non-TEE nodes due to
following reasons:

  * (1)

Most of the existing software-based attestation mechanism assumes that the
underlying platform does not support hardware TEE primitives. Such an
assumption is valid for low-power platforms such as wireless sensor networks
or embedded systems. However, in data centers, powerful nodes have massive
compute capabilities and capable of supporting TEE primitives.

  * (2)

Existing software-based attestations are limited to simpler embedded firmware,
whereas we assume a more complex, powerful device firmware that is capable of
executing complex workloads. Such a device requires a full-fledged
measurement-based attestation.

  * (3)

Existing research [117] shows software-based attestation are vulnerable to
rootkit-based and compression attacks.

Given the above-mentioned reasons, we conclude that the software-based
attestation of the node is unsuitable for data center/cloud scenario.

###  A.4 Existing Device TEEs

####  A.4.1 GPUs

Graviton [26] instantiates a TEE for NVIDIA GPUs to enable isolated execution
of kernels by using a trusted security monitor (Sm) that processes all
commands to the GPU. Graviton partitions the regions that can be mapped by the
host (for MMIO) into unprotected, protected and hidden regions. To create
these regions, it augments the PCIe controller to perform range checks that
are initialised by the SM at start up. The SM ensures that memory allocated to
a kernel is not accessible to other non-trusted entities. To guarantee memory
isolation, the SM tags pages with ids of the kernels they are allocated to and
closely tracks all page mappings. All allocation and de-allocations are
handled by the SM and the untrusted driver has no direct access to GPU memory.
However, the GPU runtime is trusted and the runtime API is augmented to enable
secure operations on the GPU.

Applications have be modified to use the secure API with the runtime. To start
a job, the driver initialises a secure context using the CH_CREATE call to the
SM. This creates a protected command channel inaccessible to the driver. This
call is also used to perform attestation, using CH_MEASURE and setup a shared
secret for channel encryption (CEK) between the host application and the GPU.
The driver uses CH_PDE to allocate memory for the secure context on the GPU.
To load a kernel and start computation, the host application uses the secure
runtime API. The runtime encrypts the kernel code and data and initiates a DMA
to transfer the encrypted data to the region it was allocated on the GPU. The
runtime then uses an Authenticated decryption kernel on the GPU to decrypt the
data and code and store it in protected private memory.

Graviton performs encrypted DMA to enclave private memory. The driver cannot
perform MMIO to the command channels as it does not have access to the CEK.
Therefore, Graviton forces the runtime to write to the command channel using
DMA. The runtime copies encrypted commands to the GPU, decrypts it using the
AuthDec kernel and then places it in the command channel. The driver can still
perform MMIO to GPU’s non-protected memory. The SM uses the unprotected space
to notify the device driver about errors, synchronization commands and
interrupts.

####  A.4.2 FPGA

ShEF [29] creates a TEE for FPGAs. Unlike computation on CPUs, FPGAs are
programmed using confidential bitstreams (binaries). ShEF ensures
confidentiality and integrity of both the bitstream and data. FPGAs deployed
in the cloud usually contain a CSP controlled Shell, that provides the API for
user bitstreams to communicate with the host.

Under the untrusted CSP threat model, ShEF isolates the user program from the
untrusted Shell. ShEF partitions the hardware into two isolated regions: one
for the Shell, and another for the trusted user bitstream. For secure boot,
ShEF requires a hardware root-of-trust on the FPGA. It also requires a
hardware SM that uses the RoT to boot the FPGA correctly and generate
attestation reports. The attestation reports are then sent via the Shell to
the user who verifies it and establishes a secure channel with the SM. The
user then sends the confidential bitstream over the secure channel to the SM.
The SM loads the user’s bitstream in the region it created and loads the
communication key into the Shield. The user can now directly communicate with
the bitstream and send data using DMA over a secure channel using the
communication key.

The Shield ensures that all data leaving the user’s bitstream region is
encrypted with the communication key. Similarly, the Shield provides a
separate interface for MMIO registers. The user can directly write encrypted
commands or data to the MMIO registers that the Shield decrypts before the
write completes. ShEF requires the developer to add a trusted Shield to their
application before the bitstream is created. This trusted Shield performs the
encryption/decryption for all data that leaves or enters the user bitstream.

###  A.5 Architecture of Ascend AI Accelerator

Figure 12: AI Accelerator Case study. An overview of the AI accelerator based
on DaVinci [70, 71] architecture that shows the core, memory subsystems ,
broadcast unit (BU), task scheduler, and the host interface.

Fig. 12 shows a simplified base model of AI accelerator with $n$ cores based
on the DaVinci architecture [70]. Each core is capable of running CNN layers
and exposes a custom ISA. The Cube Unit is responsible for matrix
multiplication and the Vector unit is responsible for vector operation. The
cube and vector units are connected to the global memory (DDR) via a unified
buffer (UB). The units fetch data through two layers of software-managed
caches $L0$ and $L1$. There are two data mover units (known as memory
transaction engine or MTE). One of the MTEs moves data between L1 and L0 and
another between L1 and the global DDR through the core front-end. The MTEs are
event-driven, i.e., MTEs move the data from the global memory to the computing
units when they need the data and writes it back to the global memory when
computation is done. The precise sequence of such operations (execute, move
data) is determined by an event scheduler on the accelerator SoC. The
scheduler is a single-stage in-order ARM core that receives the compiled
kernel instructions from the host and assigns them to the cores. The cores
rely on an in-order scalar front-end to offload instructions to the MTEs, the
Vector Unit, and the Cube Unit. The cores are connected to a broadcast unit
(BU). The BU collects memory requests from the cores and responses from the
memory subsystem and routes them to the correct endpoint. A detailed design of
the accelerator is described in [74].

Table 7: SSD configurations from SimpleSSD [81]

SSD ARM core Freq Cache size NVM controller BW Intel 750 400GB 400 MHz 800MB
3.2 GB/s Samsung Z-SSD prototype 800GB 800 MHz 1GB 4 GB/s Samsung 983 DCT
1.92TB 800 MHz 512 MB 696 MB/s Samsung 850 PRO 256GB 400 MHz 384 MB 4GB/s
Intel 535 240GB 300 MHz 256 MB 2GB/s

###  A.6 Architecture of Modern SSDs

SSDs typically contain host interface layer (HIL) that implements a interface
such as PCIe subsystem to talk to the CPU, a SSD controller, and a CPU core
(typically an ARM-M profile core) that runs the firmware of the SSD
controller. The SSD controller consists of an internal cache layer (ICL) and
page-level flash translation layer (FTL). ICL manages the DRAM cache speeds up
the IO transaction by caching recently used pages. ICL implements cache
replacement policies. FTL implements the flash translation, i.e., converting
logical page numbers to physical page numbers (both are 4K pages), garbage
collection, SSD trimming, die wear level management, and over-provisioning
management.