# VIA: Analyzing Device Interfaces of Protected Virtual Machines

Felicitas Hetzelt  TU Berlin<EMAIL_ADDRESS>,  Martin Radev
Fraunhofer AISEC<EMAIL_ADDRESS>,  Robert Buhren  TU Berlin
<EMAIL_ADDRESS>,  Mathias Morbitzer  Fraunhofer AISEC
<EMAIL_ADDRESS>and  Jean-Pierre Seifert  TU Berlin
<EMAIL_ADDRESS>
###### Abstract.

Both AMD and Intel have presented technologies for confidential computing in
cloud environments. The proposed solutions — AMD SEV (-ES, -SNP) and Intel TDX
— protect VMs against attacks from higher privileged layers through memory
encryption and integrity protection. This model of computation draws a new
trust boundary between virtual devices and the VM, which in so far lacks
thorough examination. In this paper, we therefore present an analysis of the
virtual device interface and discuss several attack vectors against a
protected VM. Further, we develop and evaluate VIA, an automated analysis tool
to detect cases of improper sanitization of input recieved via the virtual
device interface. VIA improves upon existing approaches for the automated
analysis of device interfaces in the following aspects: (i) support for
virtualization relevant buses, (ii) efficient Direct Memory Access (DMA)
support and (iii) performance. VIA builds upon the Linux Kernel Library and
clang’s libfuzzer to fuzz the communication between the driver and the device
via MMIO, PIO, and DMA. An evaluation of VIA shows that it performs 570
executions per second on average and improves performance compared to existing
approaches by an average factor of $2706$. Using VIA, we analyzed 22 drivers
in Linux 5.10.0-rc6, thereby uncovering 50 bugs and initiating multiple
patches to the virtual device driver interface of Linux. To prove our
findings’ criticality under the threat model of AMD SEV and Intel TDX, we
showcase three exemplary attacks based on the bugs found. The attacks enable a
malicious hypervisor to corrupt the memory and gain code execution in
protected VMs with SEV-ES and are theoretically applicable to SEV-SNP and TDX.

††conference: …; …; …††booktitle: …††price: …††isbn: …††submissionid: 71††ccs:
Security and privacy Virtualization and security

##  1\. Introduction

Cloud computing provides many advantages, such as on-demand resource
allocation and high-availability guarantees. Nevertheless, many enterprises
have not moved their data into the cloud due to security and privacy concerns
(Amigorena, 2019); one major concern being the compromise of the Hypervisor
(HV) which has full access to the potentially sensitive data processed within
the VM.

To protect data within cloud environments from a potentially malicious HV,
several solutions have been proposed by academia (Zhang et al., 2011; Jin et
al., 2011; Xia et al., 2013; Szefer and Lee, 2012). In 2016, AMD proposed the
first commercially available solution called Secure Encrypted Virtualization
(SEV) (Kaplan et al., 2016), followed later by similar solutions from Intel
(Intel, 2020) and IBM (Hunt et al., 2018). These solutions enhance the
security of VMs, as the VM no longer depends on the HV’s security or on the
integrity of the cloud provider. While recent research has uncovered security
issues specific to the design of the SEV technologies (Cfir Cohen, 2019;
Buhren et al., 2019; Hetzelt and Buhren, 2017; Du et al., 2017; Morbitzer et
al., 2018; Morbitzer et al., 2019; Werner et al., 2019; Li et al., 2019; Wilke
et al., 2020; Li et al., 2020), little research has been done in analyzing the
VM’s software under this new threat model (Radev and Morbitzer, 2020).
Specifically, only limited effort has been spent on analyzing security risks
due to drivers which communicate with devices controlled by the now untrusted
HV. Drivers inside the VM should now treat input from virtual devices as
malicious and should implement proper input validation. Compared to attacks
from external hardware devices against the host OS (Markettos et al., 2019;
Nohl and Lell, 2014), validation is even more critical in a protected VM as a
malicious HV has fine-grained control over the VM, greatly enhancing the
adversary’s abilities. For example, a HV is able to intercept the VM’s
execution at an arbitrary point in time and can monitor various feedback
channels to infer the execution state of the VM (Li et al., 2019).

Identification of drivers performing insufficient validation of HV controlled
input requires the analysis of a vast codebase. To be able to quickly identify
missing input validation in such a vast code base, we present VIA. VIA is a
fuzzing framework targeted to detect missing input validation in device
drivers of VMs. Although dynamic analysis tools exist for the device
interface, they require specialized hardware (Markettos et al., 2019) or
devices implementations (Song et al., 2019) and target only a limited subset
of driver functionality. Existing device independent approaches target the USB
interface (Google, 2019; Peng and Payer, 2020) and a limited subset of the
Peripheral Component Interconnect (PCI) (Song et al., 2020) interface.
Notably, none of the device independent approaches targets the complex DMA
interface of device drivers, which has been shown to be a rich source of
vulnerabilities in the past (Markettos et al., 2019). VIA supports Virtual I/O
(VirtIO), Platform and PCI device interfaces –including DMA– allowing VIA to
target a large extent of the virtual device driver interface.

In addition to limited bus support, previous approaches require a full VM
setup and suffer from poor performance due to delays embedded in device
drivers, frequent context switches between the host and the VM and inefficient
interrupt injection. To address these issues, VIA moves the fuzzing setup and
the Linux kernel into a userspace program and thereby eliminates the need for
a full VM setup including userspace and virtualization software. VIA further
increases analysis throughput by applying optimizations to the Linux kernel
environment that reduce redundant delays in driver code. To inject fuzzing
data into the virtual device interface, VIA employs in-process IO-interception
and avoids costly VMEXIT\- or pagefault-based approaches. Further, VIA
incorporates information about the target driver state to efficiently guide
the analysis through targeted interrupt injection.

Using VIA, we analyzed 22 device drivers. We identified 50 cases of missing
input validation and initiated multiple patches to the virtual device driver
interface (vmx, 2021; vir, 2021d; swi, 2021; vir, 2021e, a, f, c, b; gve,
2021b, a). In total, we found issues in 20 of the 22 analyzed drivers. To
underline the severity of the missing validations, we detail three cases of
how a malicious HV is able to execute arbitrary code or corrupt protected
memory within a VM. To evaluate VIA, we compare our approach against Agamotto
(Song et al., 2020), a state-of-the-art, VM-based, PCI device fuzzer. Our
evaluation shows that, VIA on average improves the analysis throughput by
factor of $2706$ and the percentage of driver code covered by a factor of
$2.26$.

In summary, our contributions are:

  * •

Analysis of the virtual device attack vector. We identify and describe the
virtual device interface as a new attack vector against protected VMs. This
new attack vector applies to all technologies that aim to isolate complete
commodity operating systems.

  * •

Categorization of bug classes. We identify and describe numerous bugs in the
virtual device driver interface, which we classify by type and severity. To
demonstrate the severity of the identified bugs, we describe three proof of
concept exploits to obtain code execution and corrupt memory in a protected
VM.

  * •

Targeted dynamic analysis tool. We design, implement and evaluate VIA, a
dynamic analysis tool designed to detect missing sanitization of HV-controlled
data. VIA comprises of a userspace fuzzer with improved bus support and
performance. We detail three optimizations that were applied to improve
dynamic analysis: in-process IO interception, delay reduction and targeted
interrupt injection.

##  2\. Background

The industry has proposed different security technologies that aim to protect
VMs in an untrusted environment. This work focuses on technologies for the x86
architecture: AMD’s SEV and Intel’s Trust Domain Extensions (TDX). This
section provides background on both technologies and discusses the use of
devices in virtualized environments.

###  2.1. Protected Virtual Machines

AMD SEV is the first commercially available technology aimed at protecting VMs
from a potentially malicious HV (Kaplan et al., 2016). Protection is achieved
by encrypting the VM’s memory with a secret key, that is unique to the VM and
inaccessible by the cloud provider. The memory encryption, key management and
VM-entry and -exit are handled by a separate highly privileged component: the
Platform Security Processor (PSP). Each guest controls the encryption of its
memory via a dedicated bit in its private page table. Unencrypted, also named
shared, pages can be used to exchange data with the HV. Later, AMD released
SEV-ES which enhances VM protection by encrypting and integrity-protecting the
VM’s registers. Another improvement — SEV-SNP — was announced in 2020 which
protects the emulation of special instructions, as well as the nested page
tables against manipulation. Intel recently proposed a similar technology to
SEV-SNP, called TDX (Intel, 2020). Notably, all these technologies improve
security for the protected VM through architectural changes, but do not
directly address vulnerabilities in the VM’s software.

At the time of writing, only SEV and SEV-ES are officially supported in Linux
and by available CPUs. AMD SEV-SNP and Intel TDX are not officially supported
on currently available CPUs, but patches to open-source projects have been
made public for both (Intel, 2020b; AMD, 2021).

###  2.2. Device Virtualization

While the hardware virtualization extensions provide CPU and memory
virtualization, device virtualization is performed by the HV and requires
cooperation from the VM. The VM discovers attached devices by traversing known
bus addresses and then reading the identifiers and configuration advertised by
the HV. The VM then loads and initializes the corresponding device driver. The
driver proceeds with programming the device through either regular access to
Memory-mapped I/O (MMIO) or special Programmable I/O (PIO) instructions.

While MMIO and PIO are used to send commands and receive configuration
information from the device, the communication of large messages, like network
packets, happens through DMA. A driver is responsible for allocating physical
memory buffers and providing their physical addresses to the device. When the
device decides to provide a payload to the VM, the device copies the data to
the DMA buffer and notifies the driver that new data is available. The driver
can then read the data and process it.

The transparent use of virtual devices with TDX or SEV requires changes in
guest operating systems such as Linux to accommodate for the fact that the HV
cannot read a guest VM’s private memory. At the kernel initialization stage,
the VM’s Linux kernel makes adjustments to the DMA and MMIO API implementation
to use decrypted memory for communication. The device and the corresponding
driver can use the memory for sharing frequently accessed communication
structures like buffer descriptors and ring queues without the need for memory
synchronization. Next, the swiotlb component is enabled, which allocates and
manages a large contiguous decrypted memory area. When a driver uses the
streaming DMA API to map a buffer without coherency requirements (e.g.,
dma_map_single), the implementation internally returns the physical address of
a buffer in the swiotlb decrypted memory region. When the driver synchronizes
the memory using the DMA API (e.g., dma_unmap_single), the swiotlb
implementation copies (bounces) the memory from the decrypted memory region to
the corresponding private buffer of the driver.

##  3\. Threat Model

Our threat model assumes a malicious actor who controls the HV and seeks to
compromise a protected VM. The malicious actor may have gained access to the
HV by exploiting a system vulnerability or may be a rogue server
administrator. The HV cannot directly read or modify the VM’s state, as the
VM’s memory and register contents are encrypted. The VM runs an up-to-date
operating system with a Linux kernel.

The HV manages access to hardware devices for the VM by providing the
appropriate virtualized device in the cloud setting. The virtual device can be
based on VirtIO or on a custom interface, but the VM’s Linux kernel must
support the device. The HV can configure arbitrary devices during the VM’s
boot and attach or detach any hot-pluggable device during the lifetime of the
VM.

Being in control of the virtual device, the malicious HV is able to modify any
data structure shared with the device. For example, such structures include
descriptor buffers and ring queues commonly used in many devices. The HV can
intercept MMIO and PIO operations and provide any value for MMIO reads and the
IN instruction. Additionally, the HV is able to intercept and modify any
communication between the real hardware device and the VM’s driver. For
example, the HV can modify the contents of network packets sent to the VM via
the DMA interface. Further, the HV is able to inject interrupts into the VM,
as necessary for sending a notification from a virtual device to the VM’s
kernel.

This proposed threat model precisely captures the threat models of AMD SEV
Secure Nested Paging (SEV-SNP) (Advanced Micro Devices, 2020) and Intel TDX
(Intel, 2020a). Any attacks on the proposed threat model are also applicable
to both of these technologies.

##  4\. Design

In this Section we detail the high level design of VIA. First, we give an
overview of the architecture of VIA as well as VIA’s central components.
Further, we present considerations about how the design avoids false positives
during bug discovery and describe options to extend VIA.

VIA is the first comprehensive approach to detect missing sanitization of data
provided by a virtual device to a Linux device driver. It extends the scope of
current hardware interface analysis by targeting VirtIO, PCI, and Platform
device interfaces. In addition to opening up these new interfaces for
analysis, VIA also addresses performance issues in existing approaches.

The dynamic analysis framework comprises of a feedback-driven, in-process,
userspace fuzzer that mutates data sent from a virtual device to the device
driver.

On a high level, VIA provides device simulations of varying complexity that
forward random data to the HV interface of device drivers. The drivers are
loaded as shared libraries into a userspace program that provides a Linux
kernel environment. VIA extends and modifies the Linux kernel environment to
support the analysis. Namely, VIA applies delay reduction, targeted interrupt
injection and in-process IO-interception to improve fuzzing throughput
significantly. Further, VIA extends the fuzzer’s input generation mechanism to
support the analysis and avoid deadlocks in the driver code.

Library-OS
---
Fuzzer-Engine
---
Harness
---
Device
---
Driver
---
1
---
2
---
c
---
d
---
a
---
e
---
b
---
Figure 1.  High-level overview of VIA. White arrows indicate initialization,
gray arrows indicate actions performed during each fuzzing iteration. The main
components are the Fuzzer-Engine and the Library-OS, which again contain the
Harness and the analyzed Driver that is bound to the according virtual Device.

#### System Overview

Figure 1 illustrates the overall design of VIA’s dynamic analysis framework.
The two main components are the Fuzzer-Engine and the Library-OS. The Fuzzer-
Engine is responsible for setup, monitoring, and controlling of the test
execution. It processes code coverage feedback and provides mutated data that
the analyzed Driver consumes. Additionally, it contains an exchangeable
Harness that drives interaction between the simulated Device and the driver.
The Library-OS component contains the Linux kernel code within which we
execute the driver, and software implementations for PCI, VirtIO, and Platform
devices that provide mutated data to the driver.

In order to start the analysis, the Fuzzer-Engine first starts one or more
instances of the Library-OS 1 and configures the software device of each
instance according to the selected device type 2.

During each fuzzing iteration, the Fuzzer-Engine executes the instructions
provided by the harness. The harness interacts with the Library-OS via the
standard Linux system call interface a, which we extended to facilitate
interrupt generation and device driver (un-)initialization. The role of the
harness is to issue a set of system calls to (i) initialize the driver, (ii)
trigger interactions between the driver and the device b and (iii) un-
initialize the driver. When the driver requests data from the device, the
device will either forward mutated data from the Fuzzer-Engine c or handle the
request according to its internal device simulation logic. In order to resolve
locks quickly, the Library-OS notifies the harness if a thread initiated by
the driver is waiting on a resource d; the harness then has the option to
trigger an interrupt in order to resolve the block. After each execution of
the harness, the code coverage generated by the driver is forwarded to the
Fuzzer-Engine e to guide future mutations.

#### Device Simulation

VIA simulates virtual devices in order to communicate with the HV interface of
device drivers. The Fuzzer-Engine configures the device according to a
configuration file provided by the analyst. Depending on the device type (PCI,
VirtIO, or Platform), VIA uses different approaches to simulate the device.
The device simulation generally falls into one of two categories: passthrough
and emulation.

Both passthrough and emulated devices perform an initial device specialization
based on a configuration file. The initial specialization configures device
identifiers that allow matching the driver to the device. In the case of
Platform devices, drivers are matched via their name (pla, 2021); PCI and
VirtIO devices are matched via Vendor- and Device-IDs (Group, 1998; vir,
2021g). Additionally, VIA configures the device’s resources at this stage,
namely interrupts and MMIO and PIO regions that the driver expects to be
provided by the device.

VIA uses passthrough for PCI and Platform devices. After the initial
configuration, the simulated device only forwards mutated data to the driver
and ignores data sent from the driver to the device. Interrupts have to be
triggered by the harness.

The emulation mode allows VIA to target higher layers of the driver code. To
that end, emulated devices adhere to the interface specification (e.g., (vir,
2021g)) to establish basic communication channels between driver and device.
Therefore, emulated devices process data sent from the driver to the device
and automatically trigger interrupts depending on the device’s internal state.
VIA can simulate VirtIO devices in both passthrough and emulation mode. While
passthrough simulation targets the lower levels of the VirtIO driver stack,
e.g., virtio-ring or virtio-mmio, emulation can reach code in higher layers,
including drivers such as virtio-net and virtio-blk. VIA provides a single
unified emulated VirtIO device stub to analyze arbitrary VirtIO drivers.

#### Targeted Interrupt Injection

Most driver-device interaction is initiated via the execution of interrupt
handlers that are part of the driver. The harness therefore injects interrupts
to trigger the interaction between the simulated device and the driver.
However, interrupts should not be injected randomly as this might decrease
performance as well as coverage stability. VIA addresses this issue by
providing callbacks to inject interrupts precisely when an execution thread of
the driver enters a blocked state while waiting on data from the device.

#### Delay Reduction

Driver execution is often prolonged during initialization or peripheral input
processing. This is mostly due to properties of the real hardware device that
the driver assumes to interact with. Through manual analysis of driver code,
we found these delays to be mostly redundant for software-based device
implementations. Therefore, VIA modifies common kernel interface functions to
eliminate delays in a generic way without impacting the functionality of the
driver. Note that during such delays the processor is not necessarily idle, as
the Linux kernel might schedule other workloads intermediately. Therefore,
approaches such as presented in ReVirt (Dunlap et al., 2002), that skip cpu
idle time, are not directly applicable.

#### Intercepting IO

In order to inject data into the hardware interface, we have to intercept the
normal communication path between driver and device. MMIO and PIO regions are
accessed via a defined kernel interface. VIA extends the functions comprising
this interface to inject test data during a fuzzing iteration.

For DMA, we have to distinguish between coherent and streaming DMA. Streaming
DMA is synced once before and after device interaction; therefore, the whole
DMA buffer is filled by the fuzzer at CPU synchronization points. Coherent DMA
can be accessed continuously without explicit synchronization from the guest
CPU. Therefore, fresh values should be provided at each access. However, the
kernel does not offer an interface to hook such accesses. VIA instead
facilitates the address sanitizer (LLVM, 2021) instrumentation to intercept
accesses to coherent DMA memory. Address sanitization adds instrumentation
around memory accesses that manages values stored in a shadow memory map. Upon
memory access the values in the shadow memory are modified to indicate the
allocation state of the accessed memory area. Further each access is validated
by examining the allocation state based on the values stored in the shadow
memory. VIA marks the shadow memory corresponding to coherent DMA memory areas
with a special value, which is processed by the address sanitizer error
handling code. When handling read operations, the sanitizer’s error handling
code is augmented to inject the next value from the input stream at the
faulting DMA address and then continue execution.

To improve performance, some memory access types are excluded from address
sanitization. These include accesses to allocations on the current stack frame
that are determined to be in bounds or promotable to register accesses and
accesses to certain types of global variables. We found that, for the analyzed
drivers, none of the consistent DMA allocations met these criteria.

Kernel addresses exposed to the HV can leak information about the address
space layout of the VM. To detect such issues, all values exposed to the
virtual device via MMIO, PIO or DMA are checked. If a value contains a kernel
address a warning is issued.

#### Input Generation

Each fuzzer-generated input is treated as a serialized sequence of memory
transfers from the device to the driver. Each time the driver requests data
from the device, VIA consumes the corresponding number of bytes from the test
input. If the generated input is exhausted, the input is extended with pseudo-
random data. This ensures that the driver receives different values to avoid
infinite loops and deadlocks.

#### Avoiding false positives

The adaptions of the Linux kernel environment were carefully designed to avoid
false positives during bug discovery. To that end, the framework mimics the
functionality of the swiotlb implementation in Linux, which is always enabled
with SEV and TDX. Like the Linux swiotlb implementation, VIA keeps track of
mapped DMA memory areas and checks if these areas are currently mapped before
the unmap operation can be performed. If a DMA address has not been mapped
previously the unmap invocation is rejected. Without such validation, drivers
that pass device controlled DMA addresses to dma_direct_sync_single_for_cpu or
dma_unmap_single, such as virtio-ring, sungem and sunhme, could be exploited
to corrupt the VM’s memory. Note that VIA uncovered bugs in the swiotlb
implementation, as detailed in Section 6.2. To aid the analysis, VIA handles
such issues and reports them.

While delay reduction could potentially lead to the detection of bugs that are
not exploitable if the delays are re-introduced, the HV has fine grained
control over the VM’s execution and timers (Radev and Morbitzer, 2020).
Therefore the HV can adjust scheduling and delays within the VM in order to
aid the exploitation of such bugs. Since the HV controls interrupt generation
under SEV as well as TDX, no false positives can be generated due to targeted
interrupt injection.

#### Extending VIA

VIA’s modular design allows for the automated analysis of a wide range of
different device driver types. While this work focuses on PCI, VirtIO, and
Platform devices, VIA is designed to be easily extensible to support other
device types.

Table 1. Implementation complexity of harnesses and device simulations
provided by VIA. All code is written in C. The number of lines of code is
obtained using the tool cloc (Al Danial, 2021)

Device Type | LoC
---|---
PCI | 242
Platform | 79
VirtIO | 554

Harness | LoC
---|---
Default | 20
Network | 73
Block/Char | 72

Table 1 presents the lines of code required to implement harnesses and device
simulations. Depending on the internal complexity of the simulated device, the
implementation can be very minimal as for PCI and Platform devices, or require
higher implementation effort as for VirtIO. To add a new device driver of an
already supported device type, the respective driver has to be enabled in the
Linux kernel configuration and a configuration file containing the device
identifiers has to be created. VIA already provides a default harness
implementation that performs one initialization and un-initialization of the
driver in each fuzzing iteration. This default harness performs a basic
analysis of arbitrary device drivers. Additionally, VIA provides more
elaborate harness implementations for network, block, and character drivers,
that interact with resources created by the driver and cover a larger extent
of the driver’s code base.

##  5\. Implementation

In this section, we present the implementation details for VIA. Further, we
detail the adaptions to the Linux kernel environment that enable the
functionality described in Section 4.

VIA is based upon Linux Kernel Library (LKL) (Purdila et al., 2010) and
clang’s libfuzzer (lib, 2021). The basic fuzzing infrastructure is provided by
libfuzzer, which handles mutation, test-case scheduling, as well as test-
execution and feedback. LKL provides the Linux kernel execution environment to
load and interact with device driver code.

#### Instrumentation

To collect coverage feedback, we instrument the driver code with the
recommended clang sanitizer to be used with libfuzzer: -fsanitize=fuzzer-no-
link. To avoid feedback pollution, we only enable feedback collection on the
target driver, omitting the remaining kernel code. To also enable address
sanitization instrumentation on the whole kernel, we use the clang flags
-fsanitize=address -fsanitize-recover=address. The address sanitization
provided by clang requires the C standard library (libc) to be used for heap
management. Since LKL uses the Linux kernel allocation algorithms instead of
the one provided by libc, we adapted the kernel address sanitizer to directly
modify the shadow memory maintained by clang’s userspace address sanitizer
instead of maintaining its own separate shadow memory region.

#### Delay Reduction

In order to avoid unnecessary delays while fuzzing, the kernel’s *delay,
*sleep, schedule_timeout[_*], and time_before/after functions are modified
when they are invoked from driver code. In the case of schedule_timeout,
*delay, and *sleep the framework returns immediately.

In the case of time_before and time_after, the framework always returns that
the queried time is after the specified time stamp. Timeouts, used in
wait_event_* functions, are not disabled because they are necessary for the
functionality of the driver. Additionally, the delay in queue_delayed_work is
always zeroed-out.

#### Lock Tracking

To track blocked threads waiting for events, the functions
wait_for_completion_* and *_wait_event_* are modified. Specifically, the
functions are extended to store and remove handles to the blocked thread
before and after completion of the event. Whenever a new entry is added, the
harness is notified.

##  6\. Evaluation

In this section, we first categorize the bugs uncovered in the virtual device
interface and present three exploits based on the discovered bugs. Finally, we
evaluate the performance of VIA.

###  6.1. Bug Overview

Table 2. List of examined drivers with their type, communication interface and
number of discovered bugs. QVD stands for QEMU Virtual Device, and CVM for
Google Confidential VM. We distinguish between low and high severity bugs. An
analysis of each bug can be found in our repository (fuz, 2021).

Driver | Usage | Type | Bus | #Bugs
---|---|---|---|---
Low | High
virtio_ring | QVD | Common | VIO | 2 | 4
virtio_net | QVD | NET | VIO | 1 | 1
virtio_blk | QVD | BLK | VIO | 5 | 1
virtio_crypto | QVD | CRYPTO | VIO | 2 | 0
virtio_rng | QVD | RNG | VIO | 0 | 1
virtio_console | QVD | MISC | VIO | 1 | 0
virtio_balloon | QVD | MISC | VIO | 2 | 1
virtio_input | QVD | MISC | VIO | 0 | 0
rocker | QVD | NET | PCI | 1 | 3
sungem | QVD | NET | PCI | 1 | 0*
sunhme | QVD | NET | PCI | 2 | 0*
8139cp | QVD | NET | PCI | 1 | 0
vmxnet3 | QVD | NET | PCI | 0 | 5
ne2k-pci | QVD | NET | PCI | 0 | 0
e100 | QVD | NET | PCI | 0 | 1
e1000 | QVD | NET | PCI | 0 | 3
e1000e | QVD | NET | PCI | 1 | 0
qemu_fw_cfg | QVD | MISC | PLT | 1 | 0
acpi | QVD | MISC | PLT | 0 | 1
gve | CVM | NET | PCI | 2 | 3
nvme | CVM | BLK | PCI | 1 | 1
tpm_tis | CVM | TPM | PLT | 0 | 2

  * •

* Device-controlled values passed to swiotlb_tbl_unmap_single as described in Section 6.2 were not counted as separate bugs.

VIA was used to examine a total of 22 drivers in the Linux kernel with version
5.10.0-rc6. Of the analyzed drivers, 19 belong to QEMU virtual devices and 3
to specialized hardware available in Google’s SEV-enabled confidential
computing VM (Google, 2020). The analysis was performed via intermittent runs
over the course of several months (i.e., from November 2020 to February 2021).
Each fuzzing session usually lasted between a few minutes to several hours. We
responsibly disclosed found security issues with the respective maintainers.
To prepare the drivers for the dynamic analysis, we patched problematic code
where possible by adding proper sanitization. Additionally, we removed complex
checks (e.g., checksum comparisons) on data provided by the fuzzer. This
optimization was applied to the drivers e1000, rocker and sungem. Addressing
such issues in a generic way is an orthogonal problem that has been addressed
in other works (Aschermann et al., 2019; Chen et al., 2019). The required
modifications were easy to identify from the kernel or crash log, and
workarounds could easily be developed. We list the changes to the driver code
in our public repository (fuz, 2021).

In total, VIA uncovered 50 novel bugs across the drivers listed in Table 2. At
the time of writing, 9 of the discovered bugs in swiotlb, gve, virtio_ring,
virtio_net and vmxnet3 have been, or are currently being fixed (vmx, 2021;
vir, 2021d; swi, 2021; vir, 2021e, a, f, c, b; gve, 2021b, a). The applied
patches include simple corrections of copy paste errors (vir, 2021e) or return
codes (vir, 2021d), additions of NULL pointer checks (gve, 2021a) and
corrections of driver state updates (gve, 2021b; vir, 2021f). More elaborate
patches were applied to facilitate the exlcusion of control data such as
pointers, indices or variables determining access length from device control
(vmx, 2021; swi, 2021; vir, 2021a, c, b).

Each entry in Table 2 was carefully validated by examining the source code of
the offending driver, and bug duplicates were removed. A thorough analysis of
each bug is available in our repository (fuz, 2021). The types of bugs covered
include out-of-bounds access, device-shared pointer, invalid memory access,
slab management bugs, miscellaneous bugs, assertion failure, unbound memory
allocations, and deadlocks. We distinguish between bugs of high and low
severity. We consider bugs as severe that potentially leak or corrupt guest
memory either on their own or by aiding exploitation of different
vulnerabilities. These include out-of-bounds access, device-shared pointer and
slab management bugs, such as use-after-free or double-free. The remaining bug
classes are considered low-severity, as they will impact the guest system’s
availability but likely do not undermine the security guarantees of SEV or
TDX. Low-severity bugs usually arise from implementation errors, that should
be fixed to guarantee proper functionality of the driver.

Table 3.  Distribution of bug classes found by VIA.  Bug Class | Count
---|---
Out-of-Bounds access | 14
Invalid memory access | 10
Slab management | 8
Device-shared pointer | 5
Miscellaneous | 3
Assertion failure (BUG) | 4
Unbounded allocation | 5
Deadlock | 1

Table 3 lists the bug classes and the corresponding number of discovered bugs.
Among the most common bugs are Out-of-Bounds access and Invalid memory access
bugs, with respectively $14$ and $10$ incidents. The discovered root cause of
these two classes is missing validation for the index and length fields of
shared data structures. Next are slab management bugs encompassing $8$
instances of use-after-free or double free issues. Such bugs often occur in
the teardown code of a function if error handling is incorrectly implemented.
Further, VIA uncovered $5$ device-shared pointer bugs where a VM kernel
pointer is stored in shared memory. These device-shared pointer issues were
all present in few network drivers (GVE, e100, rocker, vmxnet3). The
miscellaneous bug category comprises of $3$ unique bugs including missing
synchronization of device memory, division by zero and limited directory
traversal. Lastly, our fuzzing setup caught $4$ cases of assertions failures,
$5$ unbounded memory allocations and $1$ deadlock. The assertion failure class
includes cases when the kernel would detect an unexpected state and call
BUG(). The unbound memory allocations category considers cases for which the
kernel performs large allocations or fails to free allocated memory, leading
to out-of-memory conditions. Deadlocks can occur, if data from the device
introduces a VM state in which a kernel thread is indefinitely waiting for a
resource.

###  6.2. A Case Study of Three Bugs

In this section we discuss how a malicious administrator, with only access to
the QEMU process, can exploit three of the discovered bugs to overwrite secret
memory of a virtual machine and to gain code execution inside the virtual
machine.

The exploitation of the three bugs was performed on a virtual machine
protected with SEV Encrypted State (SEV-ES) and running Linux 5.10.8. The
exploits were designed with Kernel Address Space Layout Randomization (KASLR)
disabled, though a malicious HV can use leaked VM kernel pointers to infer the
memory layout of the VM if required.

#### Device-shared pointer in vmxnet3

⬇

1struct vmxnet3_rx_buf_info {

2 struct sk_buff *skb; // shared ptr with device

3 …

4};

5struct sk_buff {

6 void (*destructor)(struct sk_buff *skb);

7 …

8};

9void skb_release_head_state(struct sk_buff *skb) {

10 if (skb->destructor) skb->destructor(skb);

11}\end{lstlisting}

12 \caption{Vulnerable code in the \texttt{vmxnet3} driver. %

13 The pointer \texttt{skb} on Line 2 is accessible by the device and can be
overwritten.

14 By creating a fake \texttt{skb} object and overwriting the pointer, the
device can trick the driver into calling any function on Line 12.}

15\label{code:vmxnet3_desc}

16\end{figure}

17

18

19The \texttt{vmxnet3} interface relies on an array of descriptors that can be
accessed by the device.

20A \texttt{vmxnet3} descriptor, shown in Figure~\ref{code:vmxnet3_desc},
contains an associated \texttt{skb} kernel pointer among other fields.

21The pointer points to a pre-allocated \texttt{sk\_buff} object in the
\gls{VM}’s private memory, which can be used immediately when a network packet
is received.

22

23The exposure of this pointer to an untrusted device leaks information about
the VM kernel’s randomized address space and additionally allows a malicious
\gls{HV} to overwrite the pointer with a convenient value.

24When the packet is received, the contents of the \texttt{sk\_buff} are
parsed, and the object is eventually freed.

25Before freeing the packet, the \texttt{sk\_buff}’s destructor would be
called, as shown on Line 12.

26

27 In order to exploit the bug the \gls{HV}

28overwrites the \texttt{skb} pointer in the descriptor to point to decrypted
memory at the moment of sending a packet to the \gls{VM}.

29At the same time, the \gls{HV} forges a fake \texttt{sk\\_buff} object at
the new address and overwrites the destructor pointer to a suitable code
gadget in the \gls{VM}’s kernel.

30The code gadget would position the stack into decrypted memory, which would
then allow continuing execution using return-oriented programming.

31We determined empirically that many code gadgets are suitable destructor
pointers because many registers and the stack contain the address of the
\texttt{skb} in decrypted memory.

32Similar exploitation opportunities were found in the driver implementations
of \texttt{GVE}, \texttt{e100}, and \texttt{Rocker}.

33

34

35

36

37\paragraph{Use-after-Free bug in \gls{VIRTIO}-Net}

38\begin{figure}[htbp]

39 \begin{lstlisting}[style=CStyle]

40static int virtnet_probe(struct virtio_device *vdev) {

41 …

42 struct net_device *dev;

43 struct virtnet_info *vi;

44 …

45 dev = alloc_etherdev_mq(sizeof(struct virtnet_info), max_queue_pairs);

46 …

47 vi = netdev_priv(dev); // vi lies within net_device memory area

48 …

49 if (virtio_has_feature(vdev, VIRTIO_NET_F_MTU)) {

50 mtu = virtio_cread16(vdev, offsetof(struct virtio_net_config, mtu));

51 if (mtu < dev->min_mtu) { // check fails

52 goto free;

53 …

54free:

55 free_netdev(dev); // struct net_device is freed

56 return err; // err == 0

57}

58static void virtnet_remove(struct virtio_device *vdev)

59{

60 struct virtnet_info *vi = vdev->priv;

61 …

62 unregister_netdev(vi->dev); // vi->dev is device controlled

63 …

64}

Figure 2.  The HV can induce virtnet_probe to fail by providing a low mtu
value. The driver fails to set err correctly. This leads to access of the
freed memory, pointed to by vi, during device removal.

⬇

1static void e1000_dump_eeprom(struct e1000_adapter *adapter) {

2 …

3 eeprom.len = ops->get_eeprom_len(netdev); // HV controls eeprom.len

4 …

5 data = kmalloc(eeprom.len, GFP_KERNEL); // data will overlap virtnet_info

6 …

7 ops->get_eeprom(netdev, &eeprom, data); // HV controlled data is copied

8 …

9 kfree(data);

10}

Figure 3.  The function e1000_dump_eeprom is used to gain control over the
memory freed in virtnet_probe. The HV controls eeprom.len. The function
get_eeprom fills the allocated buffer with data controlled by the HV.

The virtio_net driver fails to return an error code if the initialization
fails due to an invalid mtu value provided by the HV. This leads to a use-
after-free bug when the device is removed. The vulnerable code is depicted in
Figure 2. To exploit this bug the HV first attaches the virtio_net device and
causes virtnet_probe to fail by providing a low mtu value. To obtain control
over the freed memory area, the HV then attaches the e1000 device and triggers
the code path shown in Figure 3. Specifying an eeprom.len of $0x1000$ will
cause a new allocation in the kmalloc-4k cache, which overlaps the memory area
of the freed struct virtnet_info. The structure virtnet_info contains the
field dev, which points to a net_device structure. The net_device structure
contains the function pointer ndo_uninit, that will be called in
unregister_netdev via the invocation of virtnet_remove. Similar to the
previous exploit, the HV overwrites virtnet_info->dev to point to a decrypted
memory area and sets ndo_uninit to point to a suitable code gadget. Next the
HV detaches the virtio_net device to trigger the vulnerable code path and gain
code execution.

The reliability of the exploit relies on two factors. First, the allocation of
the data allocated in e1000_dump_eeprom has to overlap the freed struct
net_device. Second, the re-allocated memory area should not be overwritten
before virtnet_remove is called. Under SEV-(-ES, -SNP) the HV has read access
to encrypted memory. This allows the HV to infer the host virtual addresses of
both allocations by monitoring changes to encrypted memory at suitable points
during the interaction between the device and the driver111Alternatively, page
fault side channels (Li et al., 2019) could be used to infer address
information.. In order to verify whether the data was overwritten, the HV
compares the encrypted memory state after e1000_dump_eeprom to the current
state before detaching the virtio_net device. If the inferred addresses differ
or the memory has been overwritten the HV can either repeat the exploit
attempt by attaching a new device, or choose to abort the attempt.

#### Out-of-Bounds bug in VirtIO/ swiotlb

Every VirtIO driver in Linux relies on a common VirtIO interface
implementation that handles the state of each communication queue. Each queue
is associated with an array of descriptors, depicted in Figure
LABEL:code:virtio_desc. When a device sends a payload to the driver, the
device typically only reads but does not update the addr and len fields since
they are provided by the driver.

⬇

1struct vring_desc {

2 __virtio64 addr;

3 __virtio32 len; // not validated.

4 __virtio16 flags;

5 __virtio16 next;

6};\end{lstlisting}

7 \caption{

8 Contents of a \gls{VIRTIO} descriptor. The device-controlled value of
\texttt{len} is not validated and is used as an argument to \texttt{memcpy}.}

9\label{code:virtio_desc}

10\end{figure}

11

12When a \gls{VM} uses AMD \gls{SEV} or Intel \gls{TDX}, the \gls{VIRTIO}
implementation relies on the DMA API and its swiotlb implementation.

13The swiotlb layer allocates a shared memory region used for communication.

14A device copies a payload to this region, the swiotlb layer performs some
validations and finally \textit{bounces} the payload to the driver’s private
buffer when memory is synchronized.

15The length of the copied data is given by the \texttt{len} field of the
\texttt{vring\\_desc} structure, shown in Figure~\ref{code:virtio_desc}.

16

17Our fuzzing tool uncovered that the swiotlb layer does not validate the
device-controlled length of the memory copy, which causes a buffer overflow.

18This allows a malicious device to overwrite the private memory of a
protected \gls{VM} with arbitrary data of arbitrary length.

19This vulnerability occurs in shared code by all Linux \gls{VIRTIO} drivers.

20What area of private memory can be overwritten depends on the memory
allocator used by the driver.

21For example, \texttt{virtio\\_console} and \texttt{virtio\\_rng} would allow
overflowing a buffer allocated from the Linux SLAB allocator.

22We verified empirically that \texttt{net\\_device} structures, e.g.
allocated during the initialization of the \texttt{e1000} device,

23are often allocated in proximity to the descriptor buffers allocated in
\texttt{virtio\\_console}.

24In such cases the \gls{HV} can exploit the out-of-bounds bug to overwrite
the contents of the structure and craft an exploit similar to the previous
example.

25

26To increase the reliability of the exploit, the \gls{HV} infers the host
virtual addresses of the descriptor buffers as well as the
\texttt{net\\_device} structures

27by observing changes in encrypted memory.

28Based on the inferred addresses, the \gls{HV} selects a descriptor that
points to a buffer that is allocated in proximity to one of the
\texttt{net\\_device} structures

29and adjusts the descriptors \texttt{len} field to overwrite the
\texttt{net\\_device} memory area.

30If no suitable locations are found the \gls{HV} re-attaches the devices
until a suitable combination of descriptor buffer and \texttt{net\\_device}
addresses is detected.

31

32

33

34\subsection{Performance Analysis}

35

36In this section, we evaluate the performance of \coolname{}.

37We compare \coolname{} against a state-of-the-art \gls{VM}-based approach.

38Further, we examine whether \textit{delay reduction} and \textit{targeted
interrupt injection} are effective in improving fuzzing performance.

39We conducted all of our experiments on a machine equipped with an 8-Core
Intel i7-7700 and 64GB of memory.

40The analysis of each driver was restricted to one CPU core, to eliminate
differences in performance due to the multithreaded design of \coolname{}.

41In order to reduce randomness, we used an empty initial corpus and a fixed
random seed for all evaluations.

42The size of individual corpus files was limited to $10$kB and a timeout of
120 seconds per fuzzing iteration was used.

43The harness modules used for the evaluation are listed in our public
repository~\cite{fuzzsarepo}.

44

45\begin{table}

46 \caption{Comparison of executions per second and percentage of driver code
covered,

47 between Agamotto~\cite{agamotto} and \coolname{} without (-D) and with
(-ND) delay reduction,

48 averaged over three hours.

49 The evaluation was performed using the basic harness module, which performs
one driver load/unload cycle.

50 The coverage is calculated as the percentage of covered \texttt{trace-pc}
instrumented instructions divided by the total number of \texttt{trace-pc}
instrumented instructions.

51 }

52 \begin{center}

53 \resizebox{\columnwidth}{!}{

54 \begin{threeparttable}

55 \begin{tabular}{l r r r @{}c r r}

56 \toprule

57 & \multicolumn{3}{c}{{\\# Executions / s}}

58 &

59 & \multicolumn{2}{c}{{\% Code Covered}}

60 \\\

61 \cmidrule{2-4}

62 \cmidrule{6-7}

63 & \thead{Agamotto} & \thead{\coolname-D} & \thead{\coolname-ND}

64 &

65 & \thead{Agamotto} & \thead{\coolname-ND}

66 \\\

67

68 \midrule

69 8139cp & 1.39 & 1336.00 & 1275.50 & & 4 & 11 \\\

70 e100 & 0.45 & 64.00 & 710.50 & & 3 & 10 \\\

71 e1000 & 0.73 & 1.00 & 66.50 & & 4 & 16 \\\

72 e1000e & 0.74 & 28.00 & 886.00 & & 2 & 9 \\\

73 gve & 1.06 & 1.00 & 732.19 & & 5 & 18 \\\

74 ne2k-pci & 0.35 & 6604.00 & 6332.00 & & 1 & 1 \\\

75 nvme & 0.79 & 0.02 & 1.55 & & 11 & 25 \\\

76 rocker & 0.64 & 2752.00 & 4227.50 & & 1 & 2 \\\

77 sungem & 0.98 & 3144.00 & 6199.50 & & 6 & 8 \\\

78 sunhme & 1.14 & 786.00 & 1250.50 & & 14 & 18 \\\

79 vmxnet3 & 0.74 & 2557.00 & 2735.00 & & 7 & 13 \\\

80 \bottomrule

81 \end{tabular}

82 \end{threeparttable}

83 }

84 \end{center}

85\label{tbl:agamotto}

86\end{table}

87

88

89To evaluate the performance improvement compared to previous \gls{VM}-based
approaches, we compare \coolname{}

90against Agamotto~\cite{agamotto}, which is a state-of-the-art \gls{PCI}
device fuzzer.

91Agamotto is similar to USBFuzz~\cite{Peng2020} in function and performance.

92It utilizes a virtual device which generates a mutated IO-stream that is

93forwarded to the target driver. In addition, Agamotto uses dynamic
checkpoints to speed up the analysis.

94The evaluation was performed using a simple harness that loads and unloads a
target driver, which is the only harness supported by Agamotto.

95Targeted interrupt injection was enabled for each driver.

96Each driver was analyzed for three hours and no modifications were applied
to the driver code.

97The results of the evaluation can be seen in Table~\ref{tbl:agamotto}.

98On average \coolname{} improves the number of executed units per second by a
factor of $1915$ and $2706$ for the delayed and non-delayed versions
respectively.

99In addition,

100we calculate the percentage of driver code covered by dividing the number
of covered \texttt{trace-pc} instrumented instructions by the

101number of all \texttt{trace-pc} instrumented instructions within the
respective driver.

102On average \coolname{} improves the percentage of covered driver code by a
factor of $2.26$.

103During the analysis \coolname{} repeatedly triggered two bugs in the
\texttt{gve} and \texttt{nvme} drivers, while Agamotto triggered none.

104

105

106\begin{table}

107 \caption{

108 Comparison of executions per second and code block coverage as reported by
\textit{libfuzzer} without (-D) and with (-ND) delay reduction,

109 averaged over three hours.

110 The evaluation was performed using the extended harness modules, which
interact with resources instantiated by the target drivers.

111 }

112 \begin{center}

113 \begin{threeparttable}

114 \begin{tabular}{l r r@{ }r @{}c r r}

115 \toprule

116 & \multicolumn{3}{c}{{\\# Executions / s}}

117 &

118 & \multicolumn{2}{c}{{\\# Blocks}}

119 \\\

120 \cmidrule{2-4}

121 \cmidrule{6-7}

122 & \thead{\coolname-D} & \multicolumn{2}{c}{\thead{\coolname-
ND\\\\\bf{(Increase)}}}

123 &

124 & \thead{\coolname-D} & \thead{\coolname-ND}

125 \\\

126 \midrule

127

128 8139cp & 1.32 & 122.41 & $\times$\bf{92.58} & & 1038 & 1040 \\\

129 acpi & 8.00 & 8.00 & $\times$1.0 & & 71 & 71 \\\

130 e100 & 63.19 & 231.98 & $\times$\bf{3.67} & & 573 & 569 \\\

131 e1000 & 3.00 & 259.06 & $\times$\bf{86.35} & & 1427 & 1535 \\\

132 e1000e & 0.70 & 111.25 & $\times$\bf{158.92} & & 1386 & 1579 \\\

133 gve & 2.00 & 636.22 & $\times$\bf{318.11} & & 147 & 594 \\\

134 ne2k-pci & 1408.00 & 1658.00 & $\times$\bf{1.18} & & 31 & 31 \\\

135 nvme & 0.02 & 0.88 & $\times$\bf{44.0} & & 260 & 291 \\\

136 qemu-fw-cfg & 1254.00 & 1341.0 & $\times$\bf{1.06} & & 35 & 37 \\\

137 rocker & 171.01 & 203.25 & $\times$\bf{1.19} & & 181 & 184 \\\

138 sungem & 6.01 & 59.04 & $\times$\bf{9.82} & & 924 & 1032 \\\

139 sunhme & 195.00 & 428.00 & $\times$\bf{2.19} & & 1025 & 1030 \\\

140 tpm-tis & 2.00 & 857.00 & $\times$\bf{428.50} & & 150 & 326 \\\

141 vio-balloon & 1291.00 & 1328.00 & $\times$\bf{1.03} & & 281 & 281 \\\

142 vio-blk & 625.00 & 624.00 & $\times$1.00 & & 333 & 333 \\\

143 vio-console & 349.00 & 444.00 & $\times$\bf{1.27} & & 352 & 352 \\\

144 vio-crypto & 270.00 & 277.00 & $\times$\bf{1.03} & & 258 & 258 \\\

145 vio-input & 393.00 & 635.00 & $\times$\bf{1.62} & & 299 & 299 \\\

146 vio-net & 553.00 & 400.00 & $\times$0.72 & & 1250 & 1257 \\\

147 vio-rng & 1.00 & 2282.00 & $\times$\bf{2282.00} & & 238 & 239 \\\

148 vmxnet3 & 37.07 & 59.94 & $\times$\bf{1.62} & & 51 & 51 \\\

149

150 \bottomrule

151 \end{tabular}

152 \end{threeparttable}%

153 \end{center}

154\label{tbl:dynamic-execs}

155\end{table}

156

157

158To evaluate the improvement through delay reduction, we measured the
difference in code block coverage and executions per second with and without
the optimization. %

159We ran one instance of \coolname{} for each driver listed in
Table~\ref{tbl:all_bugs}.

160The execution was terminated after three hours, or after $128$ restarts in
cases where bugs were triggered shortly after the start of the analysis.

161To improve the analysis results, we patched identified bugs when possible,
as detailed in our repository~\cite{fuzzsarepo}.

162For the evaluation of network drivers, we used the more elaborate harness
components for network, character and block device drivers, that are provided
by \coolname{}.

163These components load the driver, obtain the resources created by the
driver, perform a small set of system calls on the resources and unload the
driver again.

164Each harness contains a concurrent thread that blocks until a workload
initiated by the driver enters a blocked state,

165in which case the routine is woken to trigger an interrupt.

166All the \gls{VIRTIO} drivers were analyzed in \textit{simulation} mode.

167

168Table~\ref{tbl:dynamic-execs} shows the difference in execution speed
between the optimized and the un-optimized version.

169To account for differences in performance due to increased code coverage,
we additionally list the total number of covered code blocks.

170The table shows that reducing delays issued by driver code dramatically
increases performance for most drivers without a negative impact on code
coverage.

171We measure an average performance improvement by a factor of $163$ through
reducing delays.

172However, in some cases, we see either no significant improvement or even a
slight reduction in performance.

173We determined two distinct causes of this behavior.

174In the first case, we see that the code block coverage for drivers with
little or no improvement is low.

175Therefore, it is likely that the driver initialization failed early before
any major interaction with the device was performed.

176The second case covers \gls{VIRTIO} drivers.

177These drivers are specifically designed for software device back-ends and
do not rely on timeouts as much as device drivers that can also interact with
real hardware.

178We recommend disabling delay optimization in cases where it does not
increase overall performance.

179\definecolor{dodgerblue4}{rgb}{0.061,0.231,0.471}

180\begin{figure}

181 \centering

182\resizebox{\columnwidth}{!}{%

183\begin{tikzpicture}

184 \begin{axis}[

185 ybar,

186 bar shift=0.0cm,

187 xlabel={Seconds},

188 ylabel={Frequency},

189 xmin=1.0,

190 ymode=log,

191 xmode=log,

192 ymin=1,

193 y=0.3cm,

194 bar width=0.1cm,

195 ]

196\addplot[black, fill=dodgerblue4, fill opacity=1.0, draw=none] table [

197 col sep=comma,

198 y=tiq0-1-1000,

199 x=Count,

200]{./figures/data/irq.csv};

201\addlegendentry{Random}

202\addplot[black, fill=lightgray, fill opacity=0.75, draw=none] table [

203 col sep=comma,

204 y=tiq1-1-1000,

205 x=Count,

206]{./figures/data/irq.csv};

207\addlegendentry{Targeted}

208

209\end{axis}

210\end{tikzpicture}}

211 \caption{Distribution of the time in seconds required to trigger a bug in
the \texttt{rocker} driver with targeted and random interrupt injection.}

212 \label{fig:irq}

213\end{figure}

214

215We focused on the \texttt{rocker} driver to evaluate whether targeted
interrupt injection improves the bug-finding capabilities of \coolname.

216The \texttt{rocker} driver was selected for this evaluation

217because it requires multiple interrupts to be triggered during its
initialization.

218If no interrupt is triggered within a fixed time window, the initialization
fails.

219To evaluate the effectiveness of the optimization, we ran two instances of
\coolname:

220one instance used targeted interrupt injection; the other injected
interrupts at random time intervals ranging from $1-1000$ nanoseconds.

221The experiment was repeated over $1000$ runs and each run was started with
an empty corpus.

222Figure~\ref{fig:irq} compares the average time in seconds required to
trigger the first bug.

223Overall, the optimized version took on average of $6.78$ seconds to trigger
the bug, whereas the unoptimized version took $17.66$ seconds.

224Targeted interrupt injection thereby reduces the time required to trigger
the bug by a factor of $2.6$.

225

226To evaluate the overall effectiveness of \coolname{} in uncovering bugs in
the \gls{HV} interface of device drivers,

227we measured the elapsed time until the first bug is triggered and averaged
the measurement over $128$ runs.

228For this evaluation, we applied the modifications to remove complex checks
on data provided by the fuzzer as detailed above.

229In addition, we applied patches to avoid \textit{assertion failures},
\textit{unbounded allocations} and \textit{deadlocks}.

230Table~\ref{tbl:ttb} shows the results from this evaluation.

231For most drivers, the first bug is triggered within the first few minutes
of fuzzing,

232with many bugs being triggered in under one minute of analysis.

233This result demonstrates the effectiveness of \coolname{} in finding such
issues,

234but also underlines the extent of missing sanitization of data provided by
the \gls{HV} in the Linux kernel code-base.

235

236\begin{table}

237 \caption{%

238 Time required to trigger the first Bug (TTB) in seconds for each examined
driver with a known bug,

239 averaged over 128 runs.

240 Patches were applied to avoid low severity bugs due to assertion failures,
unbounded allocations and deadlocks.

241 }

242 \begin{center}

243\resizebox{\columnwidth}{!}{%

244 \begin{tabular}{l r | l r | l r}

245 \toprule

246 %

247 & \thead{TTB (s)}

248 &

249 & \thead{TTB (s)}

250 &

251 & \thead{TTB (s)} \\\

252 \midrule

253 8139cp & 4.60 & acpi & 1.87 & e100 & 1424.80 \\\

254 e1000 & 51.54 & e1000e & 1650.98 & gve & 542.82 \\\

255 nvme & 0.37 & qemu-fw & 2.03 & rocker & 6.78 \\\

256 sungem & 1.50 & tpm-tis & \textsuperscript{*}7200+ & vio-balloon & 37.86
\\\

257 vio-blk & 20.24 & vio-console & 18.28 & vio-crypto & 18.92 \\\

258 vio-net & 4.93 & vio-ring & 132.34 & vio-rng & 2.08 \\\

259 vmxnet3 & 1.89 & & & & \\\

260

261

262

263 \bottomrule

264 \end{tabular}}

265 \end{center}

266 \begin{tablenotes}

267 \item \scriptsize{\textsuperscript{*} Estimate based on few runs, since
the bug could not be triggered reliably.}

268 \end{tablenotes}

269 \label{tbl:ttb}

270\end{table}’

##  7\. Related Work

#### Hardware Interface Analysis

Work on analyzing the hardware side interface of device drivers has mainly
focused on the USB interface, with limited support for PCI and other hardware
buses. Generic approaches are usually based on symbolic execution or require
specific hardware devices.

Syzkaller-USB (Google, 2019) adapts the Linux kernel to inject USB packets via
the system call interface directly. In order to avoid the requirement for such
modifications to the target operating system, a virtual device can be used to
forward the fuzzing input. This approach is implemented by USBFuzz (Peng and
Payer, 2020) and Spenneberg et al. (Spenneberg, 2014) to analyze USB device
drivers. Agamotto (Song et al., 2020) implements a similar approach using a
virtual PCI device. Keil et al. (Keil and Kolbitsch, 2007) use a virtual
device to emulate a specific WiFi chip in order to analyze its respective
device driver. Reviewing these approaches, we found their performance to be
insufficient. Both Agamotto and USBFuzz cite an average of multiple seconds of
execution time per test case. Further, none of the existing techniques can be
applied to directly target the DMA interface and lack support for relevant
device classes such as VirtIO and Platform.

Symdrive (Renzelmann et al., 2012), POTUS (Patrick-Evans et al., 2017) and DDT
(Kuznetsov et al., 2010) facilitate the symbolic execution capabilities
provided by S2E (Chipounov et al., 2012) to systematically explore driver
state. While symbolic execution can analyze hardware interfaces in a generic
way, performance is often limited due to path explosion and expensive
constraint solving. SADA (Bai et al., 2021) proposes a static approach to
detect several classes of unsafe DMA accesses combined with lightweight
symbolic execution. While SADA improves the performance issues of pure
symbolic execution methods it incurs disadvantages due to the static nature of
the analysis. For example, bugs depending on the runtime state of the driver
such as bugs caused by improper handling of failed driver initialization are
difficult to detect with purely static approaches. Further, due to incomplete
symbolic analysis SADA produces false positives which have to be manually
analyzed.

Periscope (Song et al., 2019) presents a bus agnostic approach to intercept on
device communication between hardware and an operating system and uses this
method to implement a fuzzer. To analyze a driver Periscope requires the
corresponding device and only analyzes bottom-half interrupt handlers. Driver
(de-)initialization, interrupt generation and top-half interrupt handling are
excluded from the analysis. Markettos et al.(Markettos et al., 2019) describe
a custom hardware component that can be attached to the PCI port in order to
trigger bugs in device drivers communicating over that interface. A similar
approach based on custom hardware is presented by TTWE (van Tonder and
Engelbrecht, 2014). In comparison to these approaches, VIA does not require
specific hardware.

#### Virtualization-Free Kernel Fuzzing

Janus (Xu et al., 2019) and Hydra (Kim et al., 2019) utilize the userspace
kernel environment provided by LKL to quickly reset state during fuzzing via
fork. Based on this primitive, they build a dynamic analysis framework
targeted at finding bugs in Linux file system implementations. These
approaches, however, target file system implementations and lack support for
hardware interface analysis.

#### Attacks on SEV and TDX

Researchers have already presented security issues in AMD SEV and SEV-ES under
the threat model of a malicious HV. The discovered vulnerabilities rely on the
missing protection of Second Level Address Translation (Hetzelt and Buhren,
2017; Morbitzer et al., 2018) and missing memory integrity protection from
software attacks (Du et al., 2017; Li et al., 2019; Wilke et al., 2020). Both
issues are addressed in the designs of AMD SEV-SNP and Intel TDX. In
comparison, the vulnerabilities presented in Section 6.2 are purely software
bugs and cannot be protected against through only memory encryption and
integrity protection. The discovered bugs do not apply to a particular
confidential computing solution, but cover all solutions which rely on devices
provided by a malicious HV. Radev and Morbitzer (Radev and Morbitzer, 2020)
discovered new attack vectors for SEV-ES VMs based on trusting the HV with the
emulation of few special instructions, the creation of MMIO regions, and the
attachment of VirtIO devices. While the authors suggest that trusting
virtualized devices can impact the security of a SEV-ES VM, they only examine
the risks for the VM in delegating cryptographic operations to the device
virtio-crypto and in using the device virtio-rng as a source of entropy. Our
work also confirms the security implications for protected VMs introduced by
devices attached to the VM. However, we also provide an automated, scalable
approach for discovering software bugs in Linux drivers.

##  8\. Discussion

#### Bug Exploitability

Software bugs in a protected VM carry a high risk since a malicious HV holds
fine-grained control over the VM’s execution from the initial boot stage
(Radev and Morbitzer, 2020). For example, a malicious HV may track the VM’s
execution to execute an attack at the precise time (Radev and Morbitzer,
2020), infer address space information via side-channels (Li et al., 2019), or
employ multiple malicious virtual devices to combine bugs in various drivers.
The exploitability of the discovered bugs and exploitation steps are agnostic
to the VM protection technology, but rather depend on the VM’s Linux kernel
self-protection configuration (Kees Cook, 2017). For example, all discovered
device-shared pointer bugs are exploitable if the virtual offset of the Linux
kernel image becomes known to the attacker via another bug or side-channel.

Reducing the trusted computing base can be an effective approach for reducing
the amount of software which contains exploitable bugs and requires
validations under this new threat model. The early publicly available
implementation of Intel TDX attempts to limit the number of buggy device
drivers by adding a device filter (Intel, 2020c). The filter blocks discovery
of PCI and ACPI devices that are not present in a pre-compiled list. However,
the discovered VirtIO / swiotlb bugs from Section 6.2 are applicable to the
VirtIO devices featured in the Intel TDX device filter at the time of writing.
Additionally, cloud providers may have different device and software stack
requirements, which would require adjustments to the list of allowed devices.
For example, we determined that the Google Confidential VM does not utilize
any of the allowed devices in the TDX device filter.

#### State Accumulation

VIA avoids state accumulation by re-loading the analyzed driver after each
fuzzing iteration. However, some state might still persist and influence
future iterations. In cases where state accumulation becomes a problem the
fuzzing process can be forked as described by Xu et al. (Xu et al., 2019) to
provide a clean state for each new iteration. VIA supports libfuzzer’s Fork
Mode (lib, 2021) to periodically reset the state of the kernel environment.

#### Limitations

LKL handles concurrency by managing different posix threads that are
synchronized via semaphores. If one thread terminates or stalls its execution,
it releases the semaphore, and a new thread can be scheduled. Therefore, there
is no implicit preemption in LKL, which limits its capabilities in finding
race conditions.

As described in Section 6, VIA requires complex checks such as checksum
comparisons to be removed manually. While we do not address such issues
directly in this work, we believe that by moving the analysis to userspace,
VIA can benefit more readily from recent advances in the field. For example,
approaches based on dataflow tracing (Ryan et al., 2019) are currently only
available in userspace and can be integrated into VIA.

#### Disclosure Response

The discovered bugs were reported to several entities including AMD, Intel,
Google, driver maintainers and the Linux kernel security, network and
virtualization mailing lists. It was recommended to publicize discovered
issues, since the threat model of current Linux releases does not yet consider
malicious virtual devices. As the bugs do not affect core products of
producers and customers of protected virtual machine technologies, they need
to be addressed by driver maintainers. Several of the bugs uncovered in
swiotlb, virtio_*, vmxnet3 and gve drivers have been addressed by the
respective maintainers.

##  9\. Conclusions

We presented VIA: a framework for dynamic analysis of drivers under the threat
model of a malicious HV. VIA was applied to 22 drivers in Linux 5.10.0-rc6,
which covered network drivers, VirtIO-based drivers, and platform drivers. The
evaluation found 50 bugs from a variety of classes and initiated multiple
patches to the Linux kernel. We implement Proof-of-Concept exploits for three
bug classes that gain code execution or corrupt memory inside an AMD SEV-ES
VM. The described exploits demonstrate how the capabilities of a malicious HV,
such as control over virtual devices, fine grained control over VM execution
as well as information about the VM execution state, can be used to craft
reliable exploits. The majority of bugs were reported to the driver
maintainers, and the security implications were responsibly disclosed.

Unlike other solutions for driver analysis, VIA carefully considers the
capabilities of the HV and the semantics of the DMA API to detect more cases
of improper input sanitization and anti-patterns in device programming. VIA
provides a coverage-driven in-process userspace fuzzer built upon libfuzzer
and LKL. Compared to existing approaches, VIA reduces setup overhead by moving
the analysis into a userspace program. In combination with targeted
optimizations of the userspace kernel environment, VIA drastically improves
dynamic analysis throughput. To the best of our knowledge, VIA presents the
first approach to analyze the device driver interface under the new threat
model of protected virtual machines.

Our results suggest that many Linux device drivers do not correctly sanitize
device-provided data. Under the threat model of AMD SEV-SNP and Intel TDX,
this presents a serious security risk for the protected VM, which may be
exploited through a software driver bug. Thus, the software operating under
such a threat model should reduce the interfaces with untrusted entities and
should rely on established methods for testing the remaining interfaces.

##  10\. Acknowledgments

The authors would like to thank our shepherd, Fengwei Zhang, and the anonymous
reviewers for their valuable feedback. This material is based upon work
partially supported by the European Commission under the Horizon 2020
Programme (H2020) as part of the LOCARD project (G.A. no. 832735). Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of
our funding agencies.

## References

  * (1)
  * fuz (2021) 2021\.  Github anonymous repository: analysis of discovered bugs in device drivers.  https://github.com/fuzzsa/fuzzsa-bugs.
  * lib (2021) 2021\.  libFuzzer: a library for coverage-guided fuzz testing.  https://llvm.org/docs/LibFuzzer.html.  Accessed: 2021-21-01.
  * gve (2021a) 2021a.  Patch: gve: Add NULL pointer checks when freeing irqs.  https://github.com/torvalds/linux/commit/e96b491a0ffa35a8a9607c193fa4d894ca9fb32f.
  * gve (2021b) 2021b.  Patch: gve: Update mgmt_msix_idx if num_ntfy changes.  https://github.com/torvalds/linux/commit/5218e919c8d06279884aa0baf76778a6817d5b93.
  * swi (2021) 2021\.  Patch: swiotlb: Validate bounce size in the sync/unmap path.  https://lore.kernel.org/patchwork/patch/1364048/.
  * vir (2021a) 2021a.  Patch: Untrusted device support for virtio.  https://lists.linuxfoundation.org/pipermail/virtualization/2021-June/054685.html.
  * vir (2021b) 2021b.  Patch: virtio-ring: maintain next in extra state for packed virtqueue<EMAIL_ADDRESS>
  * vir (2021c) 2021c.  Patch: virtio-ring: store DMA metadata in desc_extra for split virtqueue.  https://lore.kernel.org/patchwork/patch/1416241/.
  * vir (2021d) 2021d.  Patch: virtio_net: Fix error code in probe().  https://lkml.org/lkml/2020/12/28/1647.
  * vir (2021e) 2021e.  Patch: virtio_ring: cut and paste bugs in vring_create_virtqueue_packed().  https://lkml.org/lkml/2020/12/28/687.
  * vir (2021f) 2021f.  Patch: virtio_ring: Fix two use after free bugs.  https://lkml.org/lkml/2020/12/28/828.
  * vmx (2021) 2021\.  Patch: vmxnet3: Remove buf_info from device accessible structures.  https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/vmxnet3?id=de1da8bcf40564a2adada2d5d5426e05355f66e8.
  * pla (2021) 2021\.  Platform Devices and Drivers.  https://www.kernel.org/doc/html/latest/driver-api/driver-model/platform.html.  Accessed: 2021-21-01.
  * vir (2021g) 2021g.  Virtual I/O Device (VIRTIO) Version 1.1.  https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html.  Accessed: 2021-21-01.
  * Advanced Micro Devices (2020) Advanced Micro Devices. 2020\.  AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More.
  * Al Danial (2021) Al Danial. 2021\.  Count Lines of Code.  https://github.com/AlDanial/cloc.  Accessed: 2021-08-06.
  * AMD (2021) AMD. 2021.  SEV-SNP OVMF.  https://github.com/AMDESE/ovmf.
  * Amigorena (2019) François Amigorena. 2019\.  Why SMBs Still do not Trust Cloud Storage Providers to Secure their Data.  https://www.infosecurity-magazine.com/opinions/smb-trust-cloud-storage-1-1/.  Accessed: 2021-21-01.
  * Aschermann et al. (2019) Cornelius Aschermann, Sergej Schumilo, Tim Blazytko, Robert Gawlik, and Thorsten Holz. 2019\.  REDQUEEN: Fuzzing with Input-to-State Correspondence.  _Ndss_ (2019).  https://doi.org/10.14722/ndss.2019.23xxx
  * Bai et al. (2021) Jia-Ju Bai, Tuo Li, Kangjie Lu, and Shi-Min Hu. 2021\.  Static Detection of Unsafe DMA Accesses in Device Drivers. In _30th USENIX Security Symposium (USENIX Security 21)_. USENIX Association.  https://www.usenix.org/conference/usenixsecurity21/presentation/bai
  * Buhren et al. (2019) Robert Buhren, Christian Werling, and Jean-Pierre Seifert. 2019\.  Insecure Until Proven Updated: Analyzing AMD SEV’s Remote Attestation. In _ACM Conference on Computer and Communications Security (CCS)_.
  * Cfir Cohen (2019) Cfir Cohen. 2019\.  AMD-SEV: Platform DH key recovery via invalid curve attack: (CVE-2019-9836).  https://seclists.org/fulldisclosure/2019/Jun/46.  Accessed: 2019-22-05.
  * Chen et al. (2019) Peng Chen, Jianzhong Liu, and Hao Chen. 2019.  Matryoshka: fuzzing deeply nested branches.  (2019).  arXiv:1905.12228
  * Chipounov et al. (2012) Vitaly Chipounov, Volodymyr Kuznetsov, and George Candea. 2012.  The S2E Platform: Design, Implementation, and Applications.  _ACM Trans. Comput. Syst._ 30, 1, Article 2 (Feb. 2012), 49 pages.  https://doi.org/10.1145/2110356.2110358
  * Du et al. (2017) Zhao-Hui Du, Zhiwei Ying, Zhenke Ma, Yufei Mai, Phoebe Wang, Jesse Liu, and Jesse Fang. 2017.  Secure Encrypted Virtualization is Unsecure.  arXiv:1712.05090 [cs.CR] https://arxiv.org/abs/1712.05090
  * Dunlap et al. (2002) George Dunlap, Samuel King, Murtaza Basrai, and Peter M Chen. 2002.  ReVirt: Enabling Intrusion Analysis through Virtual-Machine Logging and Replay.  (2002).
  * Google (2019) Google. 2019.  syzkaller - kernel fuzzer.  https://github.com/google/syzkaller.
  * Google (2020) Google. 2020.  Confidential VM and Compute Engine.  https://cloud.google.com/compute/confidential-vm/docs/about-cvm.
  * Group (1998) PCI Special Interest Group. 1998\.  _PCI Local Bus Specification_.  Chapter 6.
  * Hetzelt and Buhren (2017) Felicitas Hetzelt and Robert Buhren. 2017.  Security Analysis of Encrypted Virtual Machines. In _International Conference on Virtual Execution Environments_.
  * Hunt et al. (2018) Guerney Hunt, Richard Boivie, Eric Hall, Elaine Palmer, Dimitrios Pendarakis, and Enriquillo Valdez. 2018.  Supporting protected computing on IBM Power Architecture.  https://developer.ibm.com/articles/l-support-protected-computing/.  Accessed: 2021-21-01.
  * Intel (2020) Intel. 2020.  Architecture Specification: Intel Trust Domain Extensions (Intel TDX) Module.  https://software.intel.com/content/dam/develop/external/us/en/documents/intel-tdx-module-1eas.pdf Accessed: 2021-15-01.
  * Intel (2020a) Intel. 2020a.  Intel Trust Domain Extensions (whitepaper).
  * Intel (2020b) Intel. 2020b.  Trust Domain Extensions.  https://github.com/intel/tdx.
  * Intel (2020c) Intel. 2020c.  x86/tdx: Add device filter support for x86 TDX platform.  https://github.com/intel/tdx/commit/6789eee52aab8985e49b362379fab73aa3eecde2.
  * Jin et al. (2011) S. Jin, J. Ahn, S. Cha, and J. Huh. 2011\.  Architectural support for secure virtualization under a vulnerable hypervisor. In _2011 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. 272–283.
  * Kaplan et al. (2016) David Kaplan, Jeremy Powell, and Tom Woller. 2016\.  White Paper AMD Memory Encryption.  http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf.
  * Kees Cook (2017) Kees Cook. 2017\.  Linux Kernel Self-Protection.  https://www.usenix.org/system/files/login/articles/login_spring17_04_cook.pdf.
  * Keil and Kolbitsch (2007) Sylvester Keil and Clemens Kolbitsch. 2007.  Stateful Fuzzing of Wireless Device Drivers in an Emulated Environment.
  * Kim et al. (2019) Seulbae Kim, Meng Xu, Sanidhya Kashyap, Jungyeon Yoon, Wen Xu, and Taesoo Kim. 2019\.  Finding semantic bugs in file systems with an extensible fuzzing framework.  _Proceedings of the 27th ACM Symposium on Operating Systems Principles_ (2019).
  * Kuznetsov et al. (2010) Volodymyr Kuznetsov, Vitaly Chipounov, and George Candea. 2010\.  Testing Closed-Source Binary Device Drivers with DDT. In _Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference_ (Boston, MA) _(USENIXATC’10)_. USENIX Association, USA, 12.
  * Li et al. (2020) Mengyuan Li, Yinqian Zhang, and Zhiqiang Lin. 2020\.  CROSSLINE: Breaking ”Security-by-Crash” based Memory Isolation in AMD SEV.
  * Li et al. (2019) Mengyuan Li, Yinqian Zhang, Zhiqiang Lin, and Yan Solihin. 2019\.  Exploiting Unprotected I/O Operations in AMD’s Secure Encrypted Virtualization. In _USENIX Security Symposium_.
  * LLVM (2021) LLVM. 2021.  Address Sanitizer.  https://clang.llvm.org/docs/AddressSanitizer.html.  Accessed: 2021-08-06.
  * Markettos et al. (2019) A. Theodore Markettos, Colin Rothwell, Brett F. Gutstein, Allison Pearce, Peter G. Neumann, Simon W. Moore, and Robert N. M. Watson. 2019.  Thunderclap: Exploring Vulnerabilities in Operating System IOMMU Protection via DMA from Untrustworthy Peripherals.  February (2019).  https://doi.org/10.14722/ndss.2019.23194
  * Morbitzer et al. (2019) Mathias Morbitzer, Manuel Huber, and Julian Horsch. 2019\.  Extracting Secrets from Encrypted Virtual Machines. In _ACM Conference on Data and Application Security and Privacy (CODASPY)_.
  * Morbitzer et al. (2018) Mathias Morbitzer, Manuel Huber, Julian Horsch, and Sascha Wessel. 2018.  SEVered: Subverting AMD’s Virtual Machine Encryption. In _European Workshop on Systems Security (EuroSEC)_.
  * Nohl and Lell (2014) Karsten Nohl and Jakob Lell. 2014.  BadUSB-On Accessories That Turn Evil.  _Black Hat USA_ 1, 9 (2014), 1–22.
  * Patrick-Evans et al. (2017) James Patrick-Evans, Lorenzo Cavallaro, and Johannes Kinder. 2017.  POTUS: Probing Off-The-Shelf USB Drivers with Symbolic Fault Injection. In _11th USENIX Workshop on Offensive Technologies (WOOT 17)_. USENIX Association, Vancouver, BC.  https://www.usenix.org/conference/woot17/workshop-program/presentation/patrick-evans
  * Peng and Payer (2020) Hui Peng and Mathias Payer. 2020.  USBFuzz: A framework for fuzzing USB drivers by device emulation.  _Proceedings of the 29th USENIX Security Symposium_ (2020), 2559–2575.
  * Purdila et al. (2010) Octavian Purdila, Lucian Adrian Grijincu, and Nicolae Tapus. 2010.  LKL: The Linux kernel library.  _Proceedings - 9th RoEduNet IEEE International Conference, RoEduNet 2010_ (2010), 328–333.
  * Radev and Morbitzer (2020) Martin Radev and Mathias Morbitzer. 2020.  Exploiting Interfaces of Secure Encrypted Virtual Machines.  _arXiv preprint arXiv:2010.07094_ (2020).
  * Renzelmann et al. (2012) Matthew J. Renzelmann, Asim Kadav, and Michael M. Swift. 2012\.  SymDrive: Testing Drivers without Devices. In _Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation_ (Hollywood, CA, USA) _(OSDI’12)_. USENIX Association, USA, 279–292.
  * Ryan et al. (2019) Gabriel Ryan, Abhishek Shah, Dongdong She, Koustubha Bhat, and Suman Jana. 2019.  Fine Grained Dataflow Tracking with Proximal Gradients.  (2019).  arXiv:1909.03461
  * Song et al. (2019) Dokyung Song, Felicitas Hetzelt, Dipanjan Das, Chad Spensky, Yeoul Na, Stijn Volckaert, Giovanni Vigna, Christopher Kruegel, Jean-Pierre Seifert, and Michael Franz. 2019.  PeriScope: An Effective Probing and Fuzzing Framework for the Hardware-OS Boundary.  February (2019).  https://doi.org/10.14722/ndss.2019.23176
  * Song et al. (2020) Dokyung Song, Felicitas Hetzelt, Jonghwan Kim, Brent ByungHoon Kang, Jean-Pierre Seifert, and Michael Franz. 2020.  Agamotto: Accelerating Kernel Driver Fuzzing with Lightweight Virtual Machine Checkpoints. In _29th USENIX Security Symposium (USENIX Security 20)_. USENIX Association, 2541–2557.  https://www.usenix.org/conference/usenixsecurity20/presentation/song
  * Spenneberg (2014) Ralf Spenneberg. 2014\.  Don ’ t trust your USB ! How to find bugs in USB device drivers.
  * Szefer and Lee (2012) Jakub Szefer and Ruby B. Lee. 2012.  Architectural Support for Hypervisor-secure Virtualization.  _SIGPLAN Not._ 47, 4 (March 2012), 437–450.  https://doi.org/10.1145/2248487.2151022
  * van Tonder and Engelbrecht (2014) Rijnard van Tonder and Herman Engelbrecht. 2014.  Lowering the USB Fuzzing Barrier by Transparent Two-Way Emulation. In _8th USENIX Workshop on Offensive Technologies (WOOT 14)_. USENIX Association, San Diego, CA.  https://www.usenix.org/conference/woot14/workshop-program/presentation/van-tonder
  * Werner et al. (2019) Jan Werner, Joshua Mason, Manos Antonakakis, Michalis Polychronakis, and Fabian Monrose. 2019\.  The SEVerESt Of Them All: Inference Attacks Against Secure Virtual Enclaves. In _ACM Symposium on Information, Computer and Communications Security (ASIACCS)_.
  * Wilke et al. (2020) Luca Wilke, Jan Wichelmann, Mathias Morbitzer, and Thomas Eisenbarth. 2020.  SEVurity: No Security Without Integrity - Breaking Integrity-Free Memory Encryption with Minimal Assumptions. In _IEEE Symposium on Security and Privacy (S &P)_.
  * Xia et al. (2013) Y. Xia, Y. Liu, and H. Chen. 2013.  Architecture support for guest-transparent VM protection from untrusted hypervisor and physical attacks. In _2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA)_. 246–257.  https://doi.org/10.1109/HPCA.2013.6522323
  * Xu et al. (2019) Wen Xu, Hyungon Moon, Sanidhya Kashyap, Po-Ning Tseng, and Taesoo Kim. 2019.  Fuzzing File Systems via Two-Dimensional Input Space Exploration.  _2019 IEEE Symposium on Security and Privacy (SP)_ (2019), 818–834.
  * Zhang et al. (2011) Fengzhe Zhang, Jin Chen, Haibo Chen, and Binyu Zang. 2011\.  CloudVisor: Retrofitting protection of virtual machines in multi-tenant cloud with nested virtualization.  _SOSP’11 - Proceedings of the 23rd ACM Symposium on Operating Systems Principles_ , 203–216.  https://doi.org/10.1145/2043556.2043576