# Skyhook: Towards an Arrow-Native
Storage System

Jayjeet Chakraborty
Alexandru Uta  University of California, Santa Cruz
Santa Cruz, CA, USA
<EMAIL_ADDRESS>Leiden University
Leiden, Netherlands
<EMAIL_ADDRESS>Ivo Jimenez
Jeff LeFevre
University of California, Santa Cruz
Santa Cruz, CA, USA
<EMAIL_ADDRESS>University of California, Santa Cruz
Santa Cruz, CA, USA
<EMAIL_ADDRESS>Sebastiaan Alvarez Rodriguez
Carlos Maltzahn  Leiden University
Leiden, Netherlands
<EMAIL_ADDRESS>University of California, Santa
Cruz
Santa Cruz, CA, USA
<EMAIL_ADDRESS>
###### Abstract

With the ever-increasing dataset sizes, several file formats such as Parquet,
ORC, and Avro have been developed to store data efficiently, save the network,
and interconnect bandwidth at the price of additional CPU utilization.
However, with the advent of networks supporting $25$-$100$ Gb/s and storage
devices delivering $1,000,000$ reqs/sec, the CPU has become the bottleneck
trying to keep up feeding data in and out of these fast devices. The result is
that data access libraries executed on single clients are often CPU-bound and
cannot utilize the scale-out benefits of distributed storage systems. One
attractive solution to this problem is to offload data-reducing processing and
filtering tasks to the storage layer. However, modifying legacy storage
systems to support compute offloading is often tedious and requires an
extensive understanding of the system internals. Previous approaches re-
implemented functionality of data processing frameworks and access libraries
for a particular storage system, a duplication of effort that might have to be
repeated for different storage systems.

This paper introduces a new design paradigm that allows extending programmable
object storage systems to embed existing, widely used data processing
frameworks and access libraries into the storage layer with no modifications.
In this approach, data processing frameworks and access libraries can evolve
independently from storage systems while leveraging distributed storage
systems’ scale-out and availability properties. We present Skyhook, an example
implementation of our design paradigm using Ceph, Apache Arrow, and Parquet.
We provide a brief performance evaluation of Skyhook and discuss key results.

###### Index Terms:

programmable storage, computational storage, storage systems, data management,
distributed systems, data processing systems

Figure 1: High-level design of Skyhook.

##  I Introduction

Over the last decade, a variety of distributed data processing frameworks like
Spark [1] and Hadoop [2] have come into existence. These frameworks were built
to efficiently query vast quantities of semi-structured data and get insights
quickly. Unlike standard relational database management systems (RDBMS) such
as MySQL [3], which are optimized to manage both data storage and processing,
these systems were designed to read data from a wide variety of data sources,
including those in the cloud like S3 [4]. These systems depend on different
file formats like Parquet [5], Avro [6], and ORC [7] for efficiently storing
and accessing data. Since storage devices have been the primary bottleneck for
data processing systems for a long time, the main focus of these file formats
has been to store data efficiently in a binary format and reduce the amount of
disk I/O required to fetch the data. However, with recent advancements in
storage devices with NVMe [8] drives and network devices with Infiniband
networks [9], the bottleneck has shifted from the storage devices to the
client machine’s CPUs, rendering the notion of ”A fast CPU and slow disk”
invalid, as shown by Trivedi et al. [10]. The serialized and compressed nature
of these file formats makes reading them CPU-bound in systems with high-speed
network and storage devices, resulting in severely reduced scalability.

An attractive solution to this problem is to offload any computation to the
storage layer to achieve scalability, faster queries, and reduced network
traffic. Several popular distributed data processing systems have explored
this approach, e.g. IBM Netazza [11], Oracle Exadata [12], Redshift [13], and
PolarDB [14]. Most of these systems are built following a clean-slate approach
and use specialized and costly hardware, such as Smart SSDs [15] and FPGAs
[16] for table scanning. Building systems like these requires in-depth
understanding and expertise in using modern hardware for building database
systems. Also, modifying existing systems like MariaDB [17], as in the case of
YourSQL [18], requires modifying code that is hardened over the years which
may result in performance, security, and reliability issues. A possible way to
mitigate these issues is to have programmable storage systems with low-level
extension mechanisms that allow implementing application-specific data
manipulation and access in their I/O path. Customizing storage systems via
plugins results in minimal implementation overhead and increases the
maintainability of the software.

Programmable object storage systems such as Ceph [19, 20], Swift [21], and
DAOS [22] often provide a POSIX filesystem interface for reading and writing
files which are mostly built on top of object storage access libraries such as
“ librados ” in Ceph and “ libdaos ” in DAOS. Being programmable, these
systems provide plugin-based extension mechanisms that allow direct access and
manipulation of objects within the storage layer. We leverage these features
of programmable storage systems and develop a new design paradigm that allows
embedding widely-used data access libraries inside the storage layer. As shown
in Figure 1, the extensions on the client and storage layers allow an
application to execute data access library operations either on the client or
via the direct object access layer in the storage server.

We implement Skyhook, an instantiation of our design paradigm using RADOS [23]
as the programmable object storage backend, CephFS [24] as the POSIX layer,
Apache Arrow [25] as the data access library, and Parquet as the file format.
We evaluate the performance of Skyhook by scaling out the cluster measuring
metrics such as query latency, CPU utilization, and network traffic. The
evaluations show that Skyhook scales by offloading CPU usage for common data
processing tasks to the storage layer, freeing the client for other processing
tasks. Additionally, since compute units are co-located with storage nodes in
Skyhook, the crash recovery and consistency semantics of the storage layer
apply naturally to the query processing layer, and Skyhook queries become
fault-tolerant. Since Skyhook uses Arrow heavily, we contributed our source
code to the upstream Apache Arrow open-source project [26] [27].

In summary, our primary contributions are as follows:

  * •

A design that embeds a widely-used data access library inside storage servers
using a plugin infrastructure that does not require any changes to the data
access library and has no impact on the storage system’s resilience and
failure management.

  * •

A design that extends a widely-used data access library with a plugin that
allows offloading tasks from clients to storage servers by leveraging existing
and unmodified filesystem and object storage interfaces (assuming an
extensible object storage system).

  * •

A brief analysis of the performance gain achieved by offloading dataset scans
to the storage servers. We demonstrate that offloading dataset scan operations
to the storage layer results in faster queries, enhanced scalability, and
reduced network traffic.

##  II Background

###  II-A Ceph

Ceph is an open-source software-defined storage platform that implements a
distributed object storage layer and provides $3$-in-$1$ interfaces for
object, block, and file-level storage. One of the main properties of Ceph is
that it does not have a single point of failure because of its CRUSH [28] map
feature. On a high-level, a CRUSH map contains object-OSD mappings, which are
downloaded to the client when connecting to a Ceph cluster. The client uses
the mapping information to calculate the location of an object in the cluster
and accesses the object directly. The object storage layer of Ceph, RADOS, is
programmable as Ceph provides a plugin-based extension mechanism with its
Object Class SDK [29] that allows extending the object storage layer with
custom plugins written in languages such as C++ and Lua. These plugins are
embedded in the form of shared libraries inside Ceph OSDs and provide the
ability to access and manipulate RADOS objects on the fly within the RADOS I/O
path. The Object Class SDK offers a small subset of POSIX APIs such as read ,
write , and stat . Several Ceph components, such as the RADOS Gateway (RGW)
and the RADOS Block Device (RBD), use the Ceph Object Class SDK. We refer to
this SDK as the Ceph CLS in the rest of the paper.

Figure 2: Architecture of Skyhook.

###  II-B Apache Arrow

Arrow is a zero-copy columnar in-memory format for storing structured data
optimized for efficient analytics operations on modern hardware. Arrow’s
uniform disk, memory, and wire format allow applications to overcome any
serialization and deserialization overhead while transferring Arrow data
between different processes and systems. Apart from being a highly-efficient
in-memory format, Arrow is also a collection of several modular data
processing components, collectively utilizing which allows building parts or
all of a data processing system. Some of the data-processing pieces that Arrow
provides are as follows: Gandiva, A LLVM-based expression compiler [30];
Flight, A gRPc-based data transport interface [31]; Dataset API, A datasets
abstraction [32]; Arrow IPC, A binary data transfer protocol. Popular data
processing systems such as Spark, Dask [33], and Ray [34] support the Arrow
in-memory format.

###  II-C Apache Parquet

Parquet is an open-source columnar file format for efficient and long-term
storage of tabular data. Parquet supports several encoding schemes such as
Dictionary and Run-Length Encoding (RLE) and compression schemes such as
Snappy, LZ4, and ZSTD, which makes it highly space-efficient. One of the
essential features of Parquet is its ability to push down predicates. Parquet
files contain row group statistics and column offsets in their footer
metadata, allowing Parquet file readers to skip over row groups and column
data pages containing non-relevant data. In this way, using Parquet for data
storage greatly minimizes the I/O required for reads. These characteristics of
Parquet make it the de-facto choice for most data processing systems in the
Hadoop ecosystem.

##  III Design and Implementation

In this section, we discuss our design paradigm, the motivation behind it and
provide an in-depth description of the internals of Skyhook. Additionally, we
also discuss the portability of our design paradigm to other storage systems
and data access libraries.

###  III-A Design Paradigm

One of the most critical aspects of our design is that it allows building in-
storage data processing systems with minimal implementation effort. Our design
extends the client and storage layers of programmable object storage systems
with widely-used data access libraries requiring absolutely no modifications.
We achieve this by (1) creating a filesystem shim in the object storage layer
so that data access libraries embedded in the storage layer can continue to
operate on files, (2) mapping client requests-to-be-offloaded directly to
objects using filesystem striping metadata, and (3) mapping files to logically
self-contained fragments by using standard filesystem striping. As shown in
Figure 2, we developed one instantiation of our design paradigm using Ceph as
the storage system, Arrow as the data access library, and Parquet as the file
format. We expose our implementation called Skyhook via the Arrow Dataset API
by creating a new file format abstraction named the SkyhookFileFormat , that
extends the FileFormat API [35] in Arrow. In the next section, we discuss the
internals of Skyhook in detail.

###  III-B Skyhook: An Example Implementation

Extending the Ceph Object Store. In the storage layer, using the Ceph CLS, we
create a “ scan_op ” object class method that reads a RADOS object containing
tabular data encoded in binary file formats such as Parquet and ORC, scans it
using Arrow APIs by applying the query parameters received from the client and
returns the scanned result in the form of an Arrow table. Reading from binary
files requires a filesystem shim (a filesystem-like random access interface)
to seek around in a file and read from any offset. Since the Ceph CLS does not
provide a full POSIX-like API, for example, with a “ seek ” method, we utilize
the Ceph CLS APIs to create a filesystem shim that allows seeks by keeping
track of the file pointer, thereby providing a file-like view over objects.
Arrow provides a FileFragment API that wraps a file and allows applying scan
operations to it. It takes predicates and projections as inputs, applies them
to a file, and returns the result in the form of an Arrow table. Since the
filesystem shim allows interacting with objects as files, it seamlessly plugs
into the FileFragment API.

Tunneling through CephFS. In Skyhook, files are written to Ceph using the
CephFS POSIX interface, which stripes files over fixed-size objects stored in
the RADOS layer. During the read phase, to execute the “ scan_op ” Ceph CLS
method on a RADOS object, Skyhook must first map a file to a RADOS object.
This mapping is achieved by leveraging the file striping metadata provided by
the Metadata Server (MDS) of CephFS to calculate object IDs from file names.
We implement a direct object access (DOA) API on the client-side that
facilitates this translation and invokes the Ceph CLS method “ scan_op ” on
the corresponding RADOS object. The DOA API takes as input a serialized
flatbuffer [36] which comprises the scan request and contains the parameters
as shown in Listing 1. The “ scan_op ” method is invoked with the serialized
scan request as input. Using the DOA API, the client can interact with RADOS
objects and manipulate them directly in application-specific ways while also
having a filesystem view over the objects. It is almost impossible to
implement reads without the DOA API because the dynamically generated result
data inside the storage servers is not byte-addressable, implying a filesystem
interface cannot be used.

⬇

table ScanRequest {

file_size: long;

file_format: short;

filter_expression: [ubyte];

partition_expression: [ubyte];

dataset_schema: [ubyte];

projection_schema: [ubyte];

}

Listing 1: Scan request in Skyhook.

Extending the Arrow Dataset API. The Arrow framework provides a FileFormat API
that plugs into the Dataset API [32] and allows scanning datasets in a unified
manner. It provides APIs for scanning datasets of different file formats such
as Parquet, ORC, CSV, and Feather [37]. Since Parquet is the de-facto file
format in most modern data processing systems, we use it as a baseline for our
performance evaluations. We extend the FileFormat API in Arrow to create a
SkyhookFileFormat API that leverages the DOA API to enable offloading fragment
scan operations to Ceph OSDs. The SkyhookFileFormat API serializes the query
parameters and passes them on to the DOA API for sending them to the storage
layer. This API is generic enough to offload scan operations on every file
format supported by Arrow as long as there is a Ceph CLS method to scan the
particular file format in the storage layer. This API allows client
applications using the Arrow Dataset API to offload file scan operations to
the RADOS layer in Ceph by simply changing the file format argument in the
Dataset API as shown in Listing 2.

Establishing a File Layout Design. Being very efficient in storing and
accessing data, Parquet has become the de-facto file format for popular data
processing systems like Spark and Hadoop. Since Parquet files are often
multiple gigabytes in size, a standard way to store Parquet files is to store
them in blocks as in HDFS [38, 39], where the typical block size is $128$ MB
[40]. While writing Parquet files to HDFS, each row group is stored in a
single block to prevent reading across multiple blocks when accessing a single
row group. We aim to follow a similar file layout for storing Parquet files in
Ceph so that every row group is self-contained within an object. In our
design, a Parquet file with $R$ row groups is split into $R$ Parquet files,
each containing the data and metadata from a single row group. To retain the
optimizations due to the predicate pushdown capability in Parquet, the footer
metadata and the schema of the parent Parquet file are also serialized and
written to a separate Parquet file ending with a ”.index” extension. So, for
every Parquet file containing $R$ row groups, $R+1$ small Parquet files are
written, each contained within a single RADOS object. We refer to this set of
$R+1$ Parquet files as a logical Parquet file. During the dataset discovery
phase, we discover only those files that end with a ”.index” extension and
read out the schema information from them. During the query execution phase,
for every logical Parquet file, first, the footer metadata from the index file
is read. Then, the IDs of the row groups that qualify for scanning are
calculated based on the row group statistics present in them. Finally, the row
group IDs are translated to the corresponding filenames, and then the
underlying objects of these files are scanned in parallel via the DOA API.
Similar to scanning Parquet files containing data from row groups, the index
Parquet file is also scanned using a Ceph CLS method. The index file feature
discussed here is only a design proposal and is yet to be implemented in
Skyhook. Figure 3 shows the file layout design.

⬇

# Reading from Parquet

import pyarrow.dataset as ds

format_ = "parquet"

dataset = ds.dataset(

"/dataset", format=format_

)

dataset.to_table()

# Reading from Parquet using Skyhook

import pyarrow.dataset as ds

format_ = ds.SkyhookFileFormat(

"parquet", "/ceph.conf"

)

dataset = ds.dataset(

"/dataset", format=format_

)

dataset.to_table()

Listing 2: Reading from a Parquet dataset with and without Skyhook using the
Arrow Dataset API.

###  III-C Portability

The implementation of Skyhook consists of several pluggable and reusable
components that make it feasible to port Skyhook’s design to other systems. In
this section, we discuss how the design paradigm of Skyhook can be applied to
other programmable object storage systems and data-access libraries by
specifying the requirements in both the client and the storage layers.

We start by discussing the portability of Skyhook’s design to other
programmable object storage systems. The DOA and the storage connection APIs
defined in the SkyhookFileFormat API on the client-side needs to use the
object storage system specific access library to be able to connect to the
storage layer and execute object class methods in the storage nodes. On the
storage-side, the filesystem shim defined in the storage plugin needs to use
the storage system specific object class SDK to access the corresponding
object store. Additionally, the storage system should have a POSIX interface
that allows using a user-defined stripe unit while writing files. This is
essential to ensure that every file is backed by a single object.

Skyhook’s design paradigm can be ported to other data access libraries. To
achieve this, the data access library on the client-side should have a Dataset
API to discover and instantiate a dataset abstraction over a directory of
files. It should be able to map file scan calls to object class method calls
using the DOA API. Additionally, a format-specific file writer is required to
split large files into object-sized ones and write them in the POSIX
filesystem by manipulating the stripe unit. In the storage nodes, the data
access library should be embeddable in the form of shared libraries and should
compile with the storage plugin. Most importantly, it needs to use the
filesystem shim as a data source to read from objects as files.

Figure 3: File layout design for Skyhook.

##  IV Evaluations

We performed experiments to compare the query latency, CPU utilization, and
network bandwidth usage of accessing a dataset and filtering at the client or
the storage server. The experiments were performed on CloudLab, the NSF-funded
bare-metal-as-a-service infrastructure [41]. For our experiments, we
exclusively used machines with an $8$-core Intel Xeon D-1548 $2.0$ GHz
processor (with hyperthreading enabled), $64$ GB DRAM, a $256$ GB NVMe drive,
and a $10$ GbE network interface. These bare-metal nodes are codenamed “m510”
in CloudLab. We ran our experiments on Ceph clusters with $4$, $8$, and $16$
storage nodes and a single client node. Each storage node had a single Ceph
OSD running on the NVMe drive. We configured the OSDs to use $8$ threads to
prevent contention due to hyperthreading in the storage nodes. The number of
placement groups (PGs) was increased from $128$ to $512$ while scaling out the
cluster to avoid increased lock contention. A CephFS interface was created on
a $3$-way replicated pool and was mounted in user mode using the ceph-fuse
utility.

We used a dataset with $1.2$ billion rows and $17$ columns consisting of data
from the NYC yellow taxi dataset [42] as our workload. The in-memory size of
the dataset was found to be about $155$ GB. In this work, we experimented with
$64$ MB files as we discovered that $64$ MB files produced the best results
for both with and without Skyhook cases. The dataset was generated by
replicating a $64$ MB Parquet file $460$ times. The Parquet files used were
uncompressed. The fact that reading directly from RADOS bypasses any cache
allowed us to replicate the same file without any caching-related implications
on performance. Since a row group is supposed to be self-contained within a
single object and the unit of parallelism in the Arrow Dataset API when using
Parquet is a single row group, we used Parquet files having a single row group
backed by a single RADOS object in all our experiments. Accordingly, the
underlying object size was also $64$ MB. Since the index file feature was not
available in Skyhook, we used similarly configured datasets containing only
single row group $64$ MB Parquet files for both with and without Skyhook
experiments. We used the Python version of the Arrow Dataset API for the
experiments and used Python’s ThreadPoolExecutor for launching scans in
parallel, following an asynchronous I/O model. We measured latency,
scalability, CPU usage, and network traffic of scanning a Parquet dataset with
and without Skyhook.

###  IV-A Latency

We ran queries to select $100\%$, $99\%$, $75$%, $50$%, $25$%, $10\%$, and
$1\%$ of the rows from our dataset. The queries were crafted so that the rows
always get selected from the beginning of the row group in every file. So,
$10$% of the rows indicates that the first $10$% of the rows were selected
from every file. In the $100\%$ selectivity case, Skyhook reads and returns
all the rows without applying any filters to the dataset. The IO depth at each
storage node was maintained at $2$ across all the experiments. As shown in
Figure 4, scanning with Skyhook is slower than without Skyhook in the $100\%$
scenario since there is no reduction of data sent over the network, and on top
of that, Skyhook ends up sending more data as it uses LZ4-compressed Arrow IPC
format on the wire which has a slightly larger byte size than the uncompressed
Parquet format. On scaling out from $4$ to $8$ nodes, the performance of
scans, both with and without Skyhook, improved as there were still CPU
resources available to use. But, on scaling out from $8$ to $16$ nodes, the
scan without Skyhook bottlenecked on the client CPU usage and stopped scaling
further, whereas the scans with Skyhook, being not bottlenecked, scaled out.
Skyhook maintains scalability as it can offload and distribute computation
across all the storage nodes rather than staying CPU bottlenecked on the
client. Also, the conversion from in-memory to wire format of Arrow requires
multiple small memory copies whose overhead add up to cause significant
performance degradation in Skyhook. This performance degradation is visible in
the $4$ nodes scenario because the without Skyhook scan is not bottlenecked,
and there is nothing to trade-off the conversion overhead of Skyhook scans as
in the $8$ and $16$ nodes cases.

Figure 4: Query latency variation on scaling out from $4$ to $8$ and $16$
storage nodes.

###  IV-B CPU Usage

Figure 5 shows the CPU utilization over time by the client and the storage
layers during a query execution with $100\%$ selectivity on a cluster with
$16$ storage nodes. The CPU usage has been recorded using Prometheus’s [43]
node metrics exporter. We observe that Parquet almost exhausts the client’s
CPU even with no filtering involved. This observation implies that the client
would be unable to do any other processing work, and the query performance is
bottlenecked on the client’s CPU. On the other hand, we observe that with
Skyhook, plenty of CPU is available on the client node, and there is high CPU
utilization on the storage layer. The CPU usage on the client node can be
attributed to the decompression of LZ4-compressed Arrow IPC data received as
the result of scan operations. Hence, with plenty of CPU available on the
client-side, more asynchronous threads can be launched to improve parallelism,
or the client can take other processing tasks.

Figure 5: CPU utilization of scans without (left) and with (right) Skyhook.
The thick blue line shows the CPU utilization on the client, and the thin
lines show the CPU utilization on the storage servers.

###  IV-C Network Traffic

While performing our experiments, we recorded the network traffic received on
the client node. Figure 6 shows the network throughput for query executions
with $100$%, $10$%, and $1$% selectivities on a cluster with $16$ storage
nodes. On scanning with Skyhook, the network traffic keeps reducing as we move
from lower ($100$%) to higher ($1$%) selectivities. In contrast, without
Skyhook, the network traffic always remains the same, which implies
significant bandwidth wastage. The numbers recorded show that using Skyhook
results in significant data movement reduction. The network traffic for
Skyhook is slightly higher than without Skyhook for the $100$% scenario
because the LZ4-compressed Arrow IPC format that Skyhook uses for transferring
data is slightly larger in byte size than the uncompressed Parquet binary
format, and moving that extra data becomes an overhead since, during $100$%
selectivity, all of the data is sent back in both cases.

Figure 6: Network bandwidth usage with and without Skyhook.

###  IV-D Crash Recovery

To verify the crash recovery functionality while using Skyhook, we executed a
long-running query with $100$% row selectivity and restarted all the OSDs one-
by-one in about the middle of the query execution to simulate a crash. As
shown in Figure 7, the query execution throughput resumed back to what was
before the crash as the Ceph OSDs started restarting. This result shows that
Skyhook queries are naturally fault-tolerant due to the co-location of query
execution units with fault-tolerant storage nodes.

Figure 7: Network throughput as perceived from the client in the event of all
OSDs crashing one by one. The steep drop in throughput is where all the OSDs
crashed.

##  V Related Work

Several distributed data processing systems have embraced the idea of query
offloading to the storage layer for performance improvement. The paper on the
Albis [10] file format by Trivedi et al. explored that with high-performance
storage and networks, CPUs have become the new bottleneck. Since the CPU
bottleneck on the client hampers scalability, offloading CPU to the storage
layer has become more important. Recently, S3 introduced S3 Select [44], which
allows files in either Parquet, CSV, or JSON format to be scanned inside S3
for improved query performance. Many other systems which already support
reading from S3, e.g. Spark [45], Redshift [13], Snowflake [46], and
PushdownDB [47] have started taking advantage of S3 Select. But being an IaaS
[48], the performance of S3 Select cannot be tuned, nor can it be customized
to read from file formats except the ones it supports. Systems like IBM
Netezza [11], PolarDB [14], and Ibex [49] depend on sophisticated and costly
hardware like FPGAs and Smart SSDs to perform table scanning inside the
storage layer. These systems employ hardware-software co-design techniques to
serve their specific use cases. These systems generally follow a clean-slate
approach and are built from the ground up specifically tailored for query
offloading. The recent work by Adams et al. [50] shows that near-data
processing can be enabled without requiring modifications by ensuring that
data elements do not cross shard boundaries and by sharing data layout
information with the clients. On a high level, this approach is somewhat
similar to that of Skyhook, except that the work is in its early stages, and
the authors do not provide any performance evaluation of their prototype
implementation.

In our approach, we take a programmable storage system, Ceph, and extend its
filesystem and object storage layers to allow offloading queries leveraging
the extension mechanisms it provides. Storage systems like OpenStack Swift
[21] and DAOS [22] also provide extension mechanisms via Storelets [51] and
DAOS middleware [52] respectively. We embed Arrow libraries inside the storage
layer to build the data access logic. Our approach signifies that storage
systems should provide plugin mechanisms so that they can be easily extended
to support ad-hoc functionality and do not need modification of legacy code or
require a complete rebuild.

##  VI Future Work

As discussed in Section IV-A, currently, Skyhook’s performance suffers from
the in-memory to wire serialization overhead of Apache Arrow. We believe that
employing techniques such as using RDMA for transferring data over the network
can help alleviate this serialization overhead. Implementing this would
require changes in the wire protocols of both Arrow and Ceph. On the use cases
front, we are currently working on integrating Skyhook with in-memory SQL-
based query engines such as DuckDB [53] and evaluating the performance over
industry-standard benchmarks such as TPC-H and TPC-DS. We also plan on
assessing the performance of Skyhook with modern distributed computing systems
such as Dask and Ray, as their recent addition of Dataset API support has made
their integration with Skyhook seamless. Another line of future work deals
with using Gandiva to accelerate the query processing performance of Arrow
inside the storage layer. Finally, we plan on implementing a Flight-based
communication protocol in Ceph to make the system faster and more arrow-
native.

##  VII Conclusion

This paper presents a new design paradigm that allows extending the POSIX
interface and the object storage layer of programmable object storage systems
with plugins to allow offloading compute-heavy data processing tasks to the
storage layer. We discuss how to embed different data access libraries and
processing frameworks inside plugins to build a universal data processing
engine that supports various file formats. We present an instantiation of this
design, Skyhook, implemented using Ceph for the storage system and Arrow for
the data processing layer. Currently, Skyhook supports reading Parquet files
only, but we can easily add support for other file formats since we use Arrow
as our data access library. We also discuss our file layout design for storing
Parquet files in Ceph, similar to HDFS that allows efficient querying. We
expose our implementation via a SkyhookFileFormat API, an extension of the
Arrow FileFormat API, and also contributed our code to the Apache Arrow open-
source project. Additionally, we present a brief performance evaluation of
Skyhook and demonstrate that offloading compute-heavy query execution to the
storage layer helps improve query performance by reducing data movement
significantly.

##  VIII Acknowledgements

This work was partially funded by the United States National Science
Foundation Cooperative Agreement OAC-1836650 (IRIS-HEP), CNS-1764102, and
CNS-1705021, as well as by the UC Santa Cruz Center for Research in Open
Source Software (CROSS), and through the Dutch National Science Foundation NWO
Veni grant VI.202.195. We would also like to thank the members of the Apache
Arrow community for their extensive reviews on our pull request and to
everyone who has contributed code to the project. Finally, we would like to
thank Dr. Julian Kunkel for shepherding this paper and the anonymous reviewers
for their valuable feedback.

## References

  * [1] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, I. Stoica _et al._ , “Spark: Cluster computing with working sets.” _HotCloud_ , vol. 10, no. 10-10, p. 95, 2010.
  * [2] T. White, _Hadoop: The definitive guide_. ” O’Reilly Media, Inc.”, 2012.
  * [3] “Mysql,” https://www.mysql.com/.
  * [4] Amazon, “Amazon s3,” https://aws.amazon.com/s3/, Amazon, accessed: 2020-11-16.
  * [5] “Apache parquet,” https://parquet.apache.org/, Apache Software Foundation.
  * [6] Apache, “Apache avro,” https://avro.apache.org/, Apache Software Foundation.
  * [7] “Apache orc: High-performance columnar storage for hadoop,” Apache Software Foundation, 2018.
  * [8] Q. Xu, H. Siyamwala, M. Ghosh, T. Suri, M. Awasthi, Z. Guz, A. Shayesteh, and V. Balakrishnan, “Performance analysis of nvme ssds and their implication on real world databases,” in _Proceedings of the 8th ACM International Systems and Storage Conference_ , 2015, pp. 1–11.
  * [9] G. F. Pfister, “An introduction to the infiniband architecture,” _High performance mass storage and parallel I/O_ , vol. 42, no. 617-632, p. 10, 2001\.
  * [10] A. Trivedi, P. Stuedi, J. Pfefferle, A. Schuepbach, and B. Metzler, “Albis: High-performance file format for big data systems,” in _2018 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 18)_, 2018, pp. 615–630.
  * [11] M. Singh and B. Leonhardi, “Introduction to the ibm netezza warehouse appliance,” in _Proceedings of the 2011 Conference of the Center for Advanced Studies on Collaborative Research_ , 2011, pp. 385–386.
  * [12] Oracle, “Oracle exadata,” https://www.oracle.com/engineered-systems/exadata/, Oracle.
  * [13] A. Gupta, D. Agarwal, D. Tan, J. Kulesza, R. Pathak, S. Stefani, and V. Srinivasan, “Amazon redshift and the case for simpler data warehouses,” in _Proceedings of the 2015 ACM SIGMOD international conference on management of data_ , 2015, pp. 1917–1923.
  * [14] W. Cao, Y. Liu, Z. Cheng, N. Zheng, W. Li, W. Wu, L. Ouyang, P. Wang, Y. Wang, R. Kuan _et al._ , “$\\{$POLARDB$\\}$ meets computational storage: Efficiently support analytical workloads in cloud-native relational database,” in _18th $\\{$USENIX$\\}$ Conference on File and Storage Technologies ($\\{$FAST$\\}$ 20)_, 2020, pp. 29–41.
  * [15] J. Do, Y.-S. Kee, J. M. Patel, C. Park, K. Park, and D. J. DeWitt, “Query processing on smart ssds: Opportunities and challenges,” in _Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data_ , 2013, pp. 1221–1230.
  * [16] Z. A. O. Nasri Sulaiman, M. Marhaban, and M. Hamidon, “Design and implementation of fpga-based systems-a review,” _Australian Journal of Basic and Applied Sciences_ , vol. 3, no. 4, pp. 3575–3596, 2009.
  * [17] F. Razzoli, _Mastering MariaDB_. Packt Publishing Ltd, 2014.
  * [18] I. Jo, D.-H. Bae, A. S. Yoon, J.-U. Kang, S. Cho, D. D. Lee, and J. Jeong, “Yoursql: a high-performance database system leveraging in-storage computing,” _Proceedings of the VLDB Endowment_ , vol. 9, no. 12, pp. 924–935, 2016.
  * [19] S. A. Weil, S. A. Brandt, E. L. Miller, D. D. Long, and C. Maltzahn, “Ceph: A scalable, high-performance distributed file system,” in _Proceedings of the 7th symposium on Operating systems design and implementation_ , 2006, pp. 307–320.
  * [20] Wikipedia, “Ceph (software),” https://bit.ly/3mkapHU.
  * [21] OpenStack, “Openstack swift storlet engine overview,” https://docs.openstack.org/storlets/latest/, OpenStack.
  * [22] Z. Liang, J. Lombardi, M. Chaarawi, and M. Hennecke, “Daos: A scale-out high performance storage stack for storage class memory,” in _Asian Conference on Supercomputing Frontiers_. Springer, 2020, pp. 40–54.
  * [23] S. A. Weil, A. W. Leung, S. A. Brandt, and C. Maltzahn, “Rados: a scalable, reliable storage service for petabyte-scale storage clusters,” in _Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing’07_ , 2007, pp. 35–44.
  * [24] G. Borges, S. Crosby, and L. Boland, “Cephfs: a new generation storage platform for australian high energy physics,” in _Journal of Physics: Conference Series_ , vol. 898, no. 6. IOP Publishing, 2017, p. 062015.
  * [25] A. D. Team, “Apache arrow,” https://arrow.apache.org, 10 2018.
  * [26] “Skyhook in apache arrow,” https://git.io/JDuxJ, Apache Software Foundation.
  * [27] “Skyhook: Bringing computation to storage with apache arrow,” https://arrow.apache.org/blog/2022/01/31/skyhook-bringing-computation-to-storage-with-apache-arrow/, Apache Arrow.
  * [28] S. A. Weil, S. A. Brandt, E. L. Miller, and C. Maltzahn, “Crush: Controlled, scalable, decentralized placement of replicated data,” in _SC’06: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing_. IEEE, 2006, pp. 31–31.
  * [29] “Ceph object class sdk,” https://docs.ceph.com/en/latest/rados/api/objclass-sdk/#sdk-for-ceph-object-classes, Ceph.
  * [30] “Gandiva: A llvm-based analytical expression compiler for apache arrow,” https://arrow.apache.org/blog/2018/12/05/gandiva-donation/, Apache Arrow, 2018.
  * [31] “Introducing apache arrow flight: A framework for fast data transport,” https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/, Apache Arrow, 2019.
  * [32] “Arrow dataset api,” https://arrow.apache.org/docs/python/dataset.html, Apache Software Foundation.
  * [33] M. Rocklin, “Dask: Parallel computation with blocked algorithms and task scheduling,” in _Proceedings of the 14th python in science conference_ , vol. 126. Citeseer, 2015.
  * [34] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica, “Ray: A distributed framework for emerging ai applications,” in _Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation_ , ser. OSDI’18. USA: USENIX Association, 2018, p. 561–577.
  * [35] “Arrow file format,” https://arrow.apache.org/docs/cpp/api/dataset.html#file-formats, Apache Software Foundation.
  * [36] Google, “Google flatbuffers,” https://google.github.io/flatbuffers/, Google.
  * [37] H. Wickham, “Feather: A fast on-disk format for data frames for r and python, powered by apache arrow,” https://www.rstudio.com/blog/feather/.
  * [38] D. Borthakur, “The hadoop distributed file system: Architecture and design,” _Hadoop Project Website_ , vol. 11, no. 2007, p. 21, 2007.
  * [39] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The hadoop distributed file system,” in _2010 IEEE 26th symposium on mass storage systems and technologies (MSST)_. Ieee, 2010, pp. 1–10.
  * [40] “Hdfs block size,” http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html, Apache Software Foundation.
  * [41] D. Duplyakin, R. Ricci, A. Maricq, G. Wong, J. Duerig, E. Eide, L. Stoller, M. Hibler, D. Johnson, K. Webb, A. Akella, K. Wang, G. Ricart, L. Landweber, C. Elliott, M. Zink, E. Cecchet, S. Kar, and P. Mishra, “The design and operation of CloudLab,” in _Proceedings of the USENIX Annual Technical Conference (ATC)_ , Jul. 2019, pp. 1–14. [Online]. Available: https://www.flux.utah.edu/paper/duplyakin-atc19
  * [42] “Nyc yellow taxi trip record data,” https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page.
  * [43] “Prometheus,” https://prometheus.io/.
  * [44] R. Hunt, “S3 select and glacier select – retrieving subsets of objects,” https://aws.amazon.com/blogs/aws/s3-glacier-select/, 11 2017.
  * [45] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, I. Stoica _et al._ , “Spark: Cluster computing with working sets.” _HotCloud_ , vol. 10, no. 10-10, p. 95, 2010.
  * [46] B. Dageville, T. Cruanes, M. Zukowski, V. Antonov, A. Avanes, J. Bock, J. Claybaugh, D. Engovatov, M. Hentschel, J. Huang _et al._ , “The snowflake elastic data warehouse,” in _Proceedings of the 2016 International Conference on Management of Data_ , 2016, pp. 215–226.
  * [47] X. Yu, M. Youill, M. Woicik, A. Ghanem, M. Serafini, A. Aboulnaga, and M. Stonebraker, “Pushdowndb: Accelerating a dbms using s3 computation,” in _2020 IEEE 36th International Conference on Data Engineering (ICDE)_. IEEE, 2020, pp. 1802–1805.
  * [48] R. Moreno-Vozmediano, R. S. Montero, and I. M. Llorente, “Iaas cloud architecture: From virtualized datacenters to federated cloud infrastructures,” _Computer_ , vol. 45, no. 12, pp. 65–72, 2012.
  * [49] L. Woods, Z. István, and G. Alonso, “Ibex: An intelligent storage engine with support for advanced sql offloading,” _Proceedings of the VLDB Endowment_ , vol. 7, no. 11, pp. 963–974, 2014.
  * [50] I. F. Adams, N. Agrawal, and M. P. Mesnier, “Enabling near-data processing in distributed object storage systems,” in _Proceedings of the 13th ACM Workshop on Hot Topics in Storage and File Systems_ , 2021, pp. 28–34.
  * [51] OpenStack, “Storlet’s documentation,” https://docs.openstack.org/storlets/latest/, OpenStack.
  * [52] Intel, “Daos client apis, tools and i/o middleware,” https://github.com/daos-stack/daos/blob/master/src/README.md#client-apis-tools-and-io-middleware, Intel.
  * [53] M. Raasveldt and H. Mühleisen, “Data management for data science-towards embedded analytics.” in _CIDR_ , 2020.


# In-situ dielectric Al2O3/$\beta$-Ga2O3 Interfaces Grown Using Metal-organic
Chemical Vapor Deposition

###### Abstract

High quality dielectric-semiconductor interfaces are critical for reliable
high performance transistors. We report the in-situ metalorganic chemical
vapor deposition (MOCVD) of Al2O3 on $\beta$-Ga2O3 as a potentially better
alternative to the most commonly used atomic layer deposition (ALD). The
growth of Al2O3 is performed in the same reactor as Ga2O3 using
trimethylaluminum and O2 as precursors without breaking the vacuum at a growth
temperature of 600 0C. The fast and slow near interface traps at the Al2O3/
$\beta$-Ga2O3 interface are identified and quantified using stressed
capacitance-voltage (CV) measurements on metal oxide semiconductor capacitor
(MOSCAP) structures. The density of shallow and deep level initially filled
traps (Dit) are measured using ultra-violet (UV) assisted CV technique. The
average Dit for the MOSCAP is determined to be 7.8 $\times$ 1011 cm-2eV-1. The
conduction band offset of the Al2O3/ Ga2O3 interface is also determined from
CV measurements and found out to be 1.7 eV which is in close agreement with
the existing literature reports of ALD Al2O3/ Ga2O3 interface. The current-
voltage characteristics are also analyzed and the average breakdown field is
extracted to be approximately 5.8 MV/cm. This in-situ Al2O3 dielectric on
$\beta$-Ga2O3 with improved dielectric properties can enable Ga2O3-based high
performance devices.

###### keywords:

Ga2O3, Al2O3, In-situ dielectric, MOCVD, Fixed charge, Photo CV, Interface
traps

Saurav Roy,* Adrian E. Chmielewski, Arkka Bhattacharyya, Praneeth Ranga, Rujun
Sun, Michael A. Scarpulla, Nasim Alem, and Sriram Krishnamoorthy*

S. Roy, A. Bhattacharyya, P. Ranga, Dr. R. Sun, Prof. M. A. Scarpulla, Prof.
S. Krishnamoorthy
Department of Electrical and Computer Engineering, University of Utah, SLC, UT
84112
Dr. A. E. Chmielewski, Prof. N. Alem
Department of Materials Science and Engineering, Pennsylvania State
University, University Park, Pennsylvania 16802
Email Address<EMAIL_ADDRESS><EMAIL_ADDRESS>
The rise of $\beta$-Ga2O3 as a promising material for power electronic
applications due to its extremely high breakdown field (8 MV/cm)[1, 2] and
Baliga figure of merit (BFOM) has also provided impetus for the search of an
appropriate insulator material as the gate dielectric for device applications.
For transistor applications, a gate dielectric should exhibit low leakage
currents, have low interface trap densities to achieve a controllable
threshold voltage (VT), and should also have a higher breakdown field compared
to the underlying semiconductor. Many insulators such as SiO2,[3, 4, 5, 6]
Al2O3,[7, 8] SiN,[9] and HfO2[9, 10] etc. have been studied as a gate oxide
material and passivation layers for Gallium Oxide devices. Extreme
permittivity materials such as Barium Titanate (BaTiO3) were also studied and
used as dielectric material in $\beta$-Ga2O3 based transistors and
heterojunction schottky barrier diodes.[11, 12] Complex oxides such as ternary
rare earth alloys (Y0.6Sc0.4)2O3 were also investigated by Masten et. al. as
gate dielectric for $\beta$-Ga2O3 based MOS structures.[13]

Among all the dielectric materials, Al2O3 is studied and used most extensively
for $\beta$-Ga2O3 based devices due to its compatibility with $\beta$-Ga2O3.
Chabak et. al. demonstrated various lateral MOSFETs with Al2O3 as gate
dielectric showing excellent field strength. [14, 15, 16] Li et. al. also
demonstrated $\beta$-Ga2O3 based vertical FinFET structures with outstanding
Figure of Merit (FOM) using Al2O3 as gate dielectric.[17, 18, 19] Jian et. al.
studied the effects of post deposition annealing on the interface trap density
of the ALD deposited Al2O3 on (001) $\beta$-Ga2O3.[20] In all these reported
works the deposition of Al2O3 is primarily performed using ex-situ technique
such as Atomic layer deposition (ALD), where after its growth, the gallium
oxide substrate or epilayer is taken to a different reactor for dielectric
deposition. During such a transfer step, it is hard to completely remove any
interface contamination. Using in-situ deposition of Al2O3 in the same reactor
as the underlying Ga2O3, the surface of the Ga2O3 can be kept pristine without
exposure to the ambient. The quality of the dielectric could also improve due
to the high temperature growth by metal-organic chemical vapor deposition
(MOCVD) (500 to 1000 0C), compared to ALD (100 to 300 0C). MOCVD also enables
high deposition rates, significantly higher than that of the ALD technique.
MOCVD reactor can also facilitate high temperature in-situ annealing which can
reduce the dielectric/semiconductor interface state density as well as the
improve the bulk dielectric quality. The in-situ growth of Al2O3 on both Ga-
face and N-face GaN has been studied extensively using MOCVD.[21, 22, 23, 24,
25] In this paper, we demonstrate for the first time in-situ growth of Al2O3
using MOCVD and electrical characterization of metal/in-situ
Al2O3/$\beta$-Ga2O3 vertical MOSCAPs.

Figure 1: Schematic diagram of the fabricated MOSCAP structures and the cross
sectional TEM image of the expanded region showing different layers. Figure 2:
(a) High resolution HAADF-STEM image of the Al2O3/$\beta$-Ga2O3 in the [101]
direction. (b) low magnification HAADF-STEM image of the same region with its
corresponding EDS maps of (c) Ga, (d) Al and (e) O. Figure 3: (a) HAADF-STEM
image of the Al2O3/$\beta$-Ga2O3 interface within the direction of the EDS
line scan indicated by the yellow arrow. (b) Atomic fraction extracted form
the EDS line scan.

The growth of in-situ Al2O3 layer after the epitaxial growth of $\beta$-Ga2O3
on Sn doped (010) $\beta$-Ga2O3 and the fabrication of MOSCAPs as shown in
Figure 1 to analyze the in-situ Al2O3/$\beta$-Ga2O3 interface quality is
discussed in the experimental section. To evaluate the abruptness of the
Al2O3/$\beta$-Ga2O3 interface as well as the stoichiometry of the Al2O3,
scanning transmission electron microscope imaging within a high angle annular
dark field detector (HAADF-STEM) combined with energy dispersive x-ray
spectroscopy (EDS) were used on a cross-section of the interface prepared by
focused ion beam (FEI Helios 660). HAADF-STEM images were performed using a
FEI Titan G2 60-300 STEM with an acceleration voltage of 300kV, a condenser
lens aperture of 70 µm and a probe current of about 100 pA. Figure 2 (a) shows
a HAADF-STEM image of the Al2O3/$\beta$-Ga2O3 interface in the [101]
projection in which the $\beta$-Ga2O3 substrate is brighter then the Al2O3
film due to the heavier Ga atoms. The Al2O3/$\beta$-Ga2O3 interface is found
to be abrupt with a transitional region of about 2 nm in which Ga atoms
contrast vanishes and the high resolution is lost. The abruptness of the
Al2O3/$\beta$-Ga2O3 interface was also confirmed by EDS mapping (Bruker)
performed in the same region. Figure 2 (b) shows a HAADF-STEM image of the
interface with its corresponding EDS maps of (c) Ga, (d) Al and (e) O. A
smooth transition between the two materials is observed with a small diffusion
of Al atoms at the top of the $\beta$-Ga2O3 epilayer. In addition, the
composition of Al2O3 could be extracted from the EDS mapping analysis using
the Cliff-Lorimer ratio technique detailed in the supporting information.[26]
As a result, the film reveals an uniform composition with 38$\pm$2 % of Al and
62$\pm$2 % of O and stays relatively homogeneous along the Al2O3 film as shown
by the EDS linescan of Figure 3 (b).

To characterize the trap densities, the fabricated MOSCAPs with different
thicknesses were subjected to a series of CV measurements using Keithley 4200,
shown in Figure 3. The alternating current (AC) bias and frequency used were
30 mV and 1 MHz respectively and the MOSCAPs were swept at a rate of 200
mV/sec. The dielectric constant was extracted to be 8.2-8.5 from the
accumulation capacitance on all three samples. The methodology used to extract
the trap densities is same as the technique developed by Liu et. al. [22] and
presented briefly here. The MOSCAPs were first taken from depletion (D) to
accumulation (A) and kept at accumulation for 10 minutes to fill all the empty
electron traps. After 10 minutes, they were again swept back to depletion (1st
sweep). Another CV curve is measured by sweeping the MOSCAP from depletion to
accumulation and immediately back from accumulation to depletion (2nd sweep).
The total charge (QI) within the MOSCAPs is calculated from the equation (1),

Figure 4: (a) First $\&$ (b) second sweep of the CV hysteresis plots for
MOSCAPs with three different dielectric thicknesses. The corresponding Flat
band voltage vs oxide thickness for (c) first $\&$ (d) second sweeps.

$V_{FB}=-\frac{qQ_{I}T_{ox}}{\epsilon\epsilon_{0}}+V_{FB}^{Ideal}$ (1)

where, VFB is the flat band voltage, Tox is the oxide thickness,
V${}_{FB}^{Ideal}$ is the ideal flat band voltage, $q$ is the electron charge,
$\epsilon$ is the relative permittivity of the Al2O3 layer, and $\epsilon_{0}$
is the absolute permittivity. Assuming all the charges are located at or near
the oxide/semiconductor interface, QI can be determined from the linear fit of
the VFB vs oxide thickness plot. The change in VFB for D to A and A to D
sweeps of the hysteresis plot can be attributed to the total amount of
initially empty trapped charge within the MOSCAPs as described by the equation
(2),

$\Delta V_{FB}=-\frac{q\Delta Q_{T}T_{ox}}{\epsilon\epsilon_{0}}$ (2)

Where, $\Delta$VFB is the flat band voltage difference between the D to A and
A to D sweeps and $\Delta$QT is the trapped electron density. Figure 4 (a) and
(b) shows the 1st and 2nd sweeps of the CV hysteresis plots respectively for
the MOSCAPs with three different Al2O3 thicknesses. The CV measurements were
repeatable after the 2nd sweep, overlapping with the 2nd sweep measurements.
Different forward bias voltages were applied for MOSCAPs with different Al2O3
thicknesses to maintain approximately same electric field through the oxide
($\sim$ 2 MV/cm). Figure 4 (c) and (d) shows the corresponding VFB and
$\Delta$VFB for the 1st and second sweeps. Flat band voltages were extracted
from the deflection point of the capacitance voltage plot.[27] VFB1 and VFB2
are the flat band voltages calculated from the D to A sweeps of the 1st and
2nd sweep, respectively. Fixed charges (QI1 $\&$ QI2) are determined from the
slope of the flat band voltage vs oxide thickness plot. QI1 is the fixed
charge of the MOSCAP without any bias history. So, QI2 is the total fixed near
interface charge in the same MOSCAP, but also contains trapped electrons due
to the measurement sweeps and 10 minutes of accumulation stress. Now,
$\Delta$VFB1 and $\Delta$VFB2 are the flat band voltage differences, between
each pair of D to A and A to D sweeps of the 1st and 2nd sweep respectively.
During the 10 minutes of accumulation stress of the first hysteresis
measurement electrons fill the empty slow near interface states and due to the
high emission time of the slow traps these electrons remain trapped
afterwards. However, electron will fill the fast near interface states also
during the 10 min accumulation time. So, $\Delta$VFB1 can be attributed to the
initially empty fast and slow near interface states ($\Delta$QT1). Only the
fast traps will respond during the second hysteresis and subsequent
measurements since the slow traps were filled during the first sweep. So
$\Delta$VFB2 can be attributed to only the fast near interface
states($\Delta$QT2). Density of the initially empty slow near interface traps
can be evaluated from the difference between $\Delta$QT1 and $\Delta$QT2
($\left|\Delta Q_{T1}-\Delta Q_{T2}\right|$). However, during the first D to A
and A to D sweeps, there could be electron emission from some of the slow near
interface states due to the applied reverse bias. So, the difference in flat
band voltage for the D to A sweeps of 1st and 2nd sweep can be attributed to
the more stably filled slow near interface states ($\Delta$QTS). All the fixed
and trapped charges described here are summarized in Table 1. The positive
fixed charge observed in this work is similar to the previous report of ALD
Al2O3/ (-201) Ga2O3 interface.[7]

Table 1: Fixed charge and trap densities at the Al2O3/$\beta$-Ga2O3 interface QI1 | QI2 |  $\Delta$QT1 |  $\Delta$QT2 | $\left|\Delta Q_{T1}-\Delta Q_{T2}\right|$ |  $\Delta$QTS
---|---|---|---|---|---
2.3 $\times$ 1012 cm-2 | 2 $\times$ 1012 cm-2 | -1.7 $\times$ 1012 cm-2 | -1.2 $\times$ 1012 cm-2 | 5 $\times$ 1011 cm-2 | 3 $\times$ 1011 cm-2

The conduction band offset at the Al2O3/$\beta$-Ga2O3 heterostructure can be
evaluated using equation (3) as VFB can be written as,[7]

$qV_{FB}=-qE_{ox}T_{ox}+(\phi_{b}-\Delta E_{C}-\phi_{S})$ (3)

where, Eox is the electric field in the oxide at flat band condition due to
the fixed charge at the, $\phi_{b}$ is Ni/Al2O3 conduction band barrier
height, $\phi_{S}$ the difference between the conduction band and fermi level
in $\beta$-Ga2O3, and $\Delta$EC is the conduction band offset between
$\beta$-Ga2O3 and Al2O3. $\phi_{b}$ was considered to be 3 eV as evaluated by
Zhang et. al. using internal photoemission spectroscopy.[28] From Figure 4 (c)
and (d), ($\phi_{b}-\Delta E_{C}-\phi_{S}$) is extracted to be 1.23 eV. Using
equation 3, the value of $\Delta$EC is calculated to be 1.7 eV which matches
well with the previous reports on the ALD Al2O3/$\beta$-Ga2O3 heterostructure.

Figure 5: Representative energy band diagram for the UV assisted CV
measurements showing (a) generation of electron-hole pairs when excited by UV
rays, (b) generated holes recombining with the trapped electrons at the
interface, and (c) trap states became positively charged after the electron
hole recombination process.

Now, to find the density of all the initially filled slow near interface
states and the trap distribution as a function of energy dependence near the
$\beta$-Ga2O3 band edge, we use UV assisted CV technique developed by Swenson
et. al.[29] and later modified by Liu et. al.[30] This technique was also used
by Jian et. al.[20] for ALD Al2O3/(001) $\beta$-Ga2O3 MOSCAPs and briefly
summarized here. The MOSCAPs were first taken from D to A and kept at
accumulation for 10 minutes to fill all the interface states. After 10 minutes
time the MOSCAPs were again swept back to depletion without any optical
excitation. This dark curve is considered as the ideal CV curve, assuming all
the states are filled with electrons. Next the MOSCAPs were held at depletion
and were exposed to UV light using a DUV LED with 254 nm wavelength for 5
minutes. After 5 minutes the UV light is turned off and the MOSCAPs were held
at depletion for 15 minutes followed by another sweep from D to A. The dark
curve is then shifted horizontally to match the post DUV CV curve and obtained
as the reference CV curve. The total density of the all the interface states
is calculated from the difference between the post DUV and the reference CV
curve ($\Delta$V) and their energy dependence is calculated from the amount of
band bending (dielectric/semiconductor surface potential) that occurs during
the bias sweep.

Figure 6: UV assisted CV characteristics and trap density vs energy
distribution plots for MOSCAPs with (a) $\&$ (d) 55 nm, (b) $\&$ (e) 24 nm,
(c) $\&$ (f) 14 nm Al2O3 thickness.

Figure 5 (a) to (c) shows energy band diagrams of the MOSCAPs under applied
bias and UV illumination. During the UV exposure, electron hole pairs are
generated and the photo-generated holes either recombine with the electrons
trapped in the interface or gets self-trapped, which results in the ledge
observed in the post DUV curve. Due to the recombination of photo generated
holes with the trapped electrons at the Al2O3/$\beta$-Ga2O3 interface, the
trapped states become positively charged, which changes the width of the
depletion region. Some of the extra holes also get trapped in the dielectric
bulk hole traps inside the Al2O3 layer. Figure 6 (a) to (c) shows CV plots for
the MOSCAPs with three different oxide thicknesses. The black curve (Dark) is
the A to D sweep after the 10 minutes of accumulation stress. The purple curve
is the D to A sweep after DUV exposure and the dashed black curve is the
reference curve obtained after shifting the dark curve to match the post DUV
plot. $\Delta$V in Figure 6 (a) to (c) includes the effects of both interface
traps at the Al2O3/$\beta$-Ga2O3 interface and bulk hole traps inside the
Al2O3 layer. Density of all the initially filled interface and bulk dielectric
traps is calculated using the equation (4),

$D_{t}=\frac{C_{ox}}{2A}\frac{d\Delta V}{d\psi_{S}}$ (4)

where, $\psi_{S}$ is the surface potential at the Al2O3/$\beta$-Ga2O3
interface, given by $\psi_{S}$ =
$\frac{q\epsilon\epsilon_{0}N_{D}A^{2}}{2C_{dep}^{2}}$. ND is the doping
density, $A$ is the area of the contacts, Cdep is the depletion capacitance
and Cox is the oxide capacitance.

Figure 6 (d) to (f) shows the Dt vs the trap energy distribution for the
MOSCAPs with three different oxide thicknesses. Due to the existence of a
positive valence band offset, the extra holes (which did not recombine with
the interface traps) get accumulated at the Al2O3/$\beta$-Ga2O3 interface
(some holes also get self-trapped). The rise in the Dt near the conduction
band edge can be attributed to the accumulated holes, either at the
Al2O3/$\beta$-Ga2O3 interface or possibly also due to the self-trapped holes
in the depletion region.[31] The average Dt for each of the MOSCAPs are
obtained after eliminating the contribution from holes by neglecting the rise
in Dt.

We can also see from Figure 6 (d) to (f) that the average Dt reduces as the
thickness of the Al2O3 layer decreases. There would not be any dielectric
thickness dependence if all the traps were located at the Al2O3/$\beta$-Ga2O3
interface. Instead, due to the presence of bulk hole traps inside the Al2O3
layer, some of the holes get captured inside the Al2O3 layer. So, the density
of the interface states (Dit) without the contribution from the bulk hole
traps in the Al2O3 is calculated from the equation (5).[30]

$D_{t}=D_{it}+T_{ox}\frac{dn_{ht}}{2d\psi_{S}}$ (5)

Where, nht is the amount of bulk hole traps inside the Al2O3. So, average Dit
is obtained to be 7.8 $\times$ 1011 cm-2 eV-1 by extrapolating the linear fit
of average Dt vs oxide thickness for Tox = 0 as shown in Figure 7. The amount
of bulk hole traps (nht) is estimated from the equation 5 to be 2.1 $\times$
1017 cm-3 eV-1. The fact that we see a linear relationship of Dt vs oxide
thickness clearly indicates that not all the holes are self-trapped in the
$\beta$-Ga2O3 and in-fact can move and get inside the Al2O3 layer and get
trapped. The mobility of holes could be significant enough to cause bulk hole
trapping in the dielectric as also extracted by Akyol [32] for $\beta$-Ga2O3
schottky barrier diode based photodetector. The value of Dit obtained for the
in-situ Al2O3 MOSCAPs are significantly lower than the previous reports of ALD
Al2O3/ (001) $\beta$-Ga2O3 interface and may have resulted from the O
vacancies and Al dangling bonds at the interface.

Figure 7: Average trap density vs oxide thickness plot and the corresponding
linear fit.

To determine leakage currents and breakdown voltages of the MOSCAPs, reverse
and forward current voltage (IV) characteristics is measured and shown in
Figure 8 (a) and (b) respectively. The measurement is performed at a sweep
rate of 200 mV/sec. Breakdown voltages with positive bias on the gate metal
(accumulation) for MOSCAPs with 14, 24 and 55 nm Al2O3 thicknesses are 9, 14,
and 32 V, respectively. The breakdown field for all the MOSCAPs were
determined using Sentaurus device simulator[33] and the field profile is shown
in Figure 9 (a). For the accumulation case with positive potential on the
gate, the entire voltage is dropped across the Al2O3 layer and so the
breakdown can be attributed to the breakdown of the dielectric. The average
breakdown field at the center of the gate metal region and peak field at the
edges of the metal for the Al2O3 dielectric are estimated to be $\sim$ 5.8
MV/cm and 15 MV/cm respectively from electric field simulations as shown in
Figure 9 (a). Electric fields were shown only in the Al2O3 region during
forward bias as the entire bias voltage is dropped across the oxide layer. The
barrier to tunneling for the MOSCAPs were also estimated using trap assisted
tunneling model and extracted to be 1.1 eV and presented in the supporting
information. The reverse breakdown voltage from the IV characteristics in
Figure 8 (a) for MOSCAPs with 14, 24 and 55 nm Al2O3 thicknesses are
approximately 40, 80, and 200 V, respectively. From the electric field
distribution in Figure 9 (b), a peak field of $\sim$ 15 MV/cm is observed at
the metal edges but the center of the gate metal, the electric field is less
than the observed field in case of forward bias ($\sim$ 3.5 MV/cm as compared
to 5.8 MV/cm). This is due to the fact that, at reverse bias, the total
applied voltage is dropped across both Al2O3 and Ga2O3 region and a
significant amount of electric field is present in the Ga2O3 ($\sim$ 2.6
MV/cm) as also shown by the dashed curves in Figure 9 (b). At reverse bias,
all the three MOSCAPs (Tox = 14, 24 and 55 nm) have a peak electric field of
around $\sim$ 15 MV/cm, indicating that the breakdown is probably happening at
the metal edges. These high edge electric fields can be reduced by creating a
field plate structure using the same in-situ Al2O3 material showing the
versatility of this technique.

Figure 8: (a) Reverse and (b) forward current voltage characteristics for
MOSCAPs with three different dielectric thicknesses. Figure 9: Simulated
electric field profiles for the MOSCAPs with three different Al2O3 thicknesses
at (a) forward breakdown volatge along A-A’(9, 14, and 32 V for Tox = 14, 24,
and 55 nm respectively) and (b) reverse breakdown volatge along A-A’ (solid
line) and B-B’ (dashed line) (-40, -80, and -200 V for Tox = 14, 24, and 55 nm
respectively). Electric field is plotted only at the Al2O3 region at forward
bias because entire bias voltage is dropped across the oxide layer during
accumulation

The in-situ Al2O3 grown using MOCVD, in this work, is found to have reduced
fixed charges and interface traps (Dit) compared to the more commonly used ALD
technique. A lower fixed charge concentration of +2 × 1012 cm-2 is found when
compared to ALD Al2O3 on (-201) $\beta$-Ga2O3 (+3.6 $\times$ 1012 cm-2)
reported by Hung et. al.[7] A lower interface trap density (Dit) of 7.8
$\times$ 1011 cm-2eV-1 is also found in comparison to ALD Al2O3/ (001)
$\beta$-Ga2O3 interface reported by Jian et. al. (1.34 $\times$ 1012
cm-2eV-1).[20] It is to be noted that no high temperature post deposition
annealing (PDA) or post metallization annealing (PMA) is performed on these
MOSCAPs, which is expected to further improve the interface quality.[34]
Further experiments to understand the effect of growth parameters such as
growth temperature, chamber pressure and precursor ratios and also higher
temperature PDA will be necessary to improve the dielectric constant, fixed
charge, interface traps and critical breakdown fields of the Al2O3 layers in
these fully MOCVD-grown Al2O3/$\beta$-Ga2O3 MOSCAPs. Also, detailed work is
necessary to investigate the origin of the interface traps which may have
resulted from native defects and impurities at the interface and can be
performed using structural and spectroscopic techniques.

In conclusion, we report in-situ Al2O3 growth using MOCVD as a potentially
better alternative to ALD and the interface quality is investigated using
capacitance voltage characterization. The presence of near interface fixed
charge in these MOSCAPs were identified and quantified using the linear
relationship between flat band voltage and oxide thicknesses and a stable
positive fixed charge of +2 $\times$ 1012 cm-2 is calculated. The sheet
density of trap states with fast and slow emission times were also calculated
from linear relationship of the CV hysteresis versus Al2O3 thickness plots.
The density of all the near interface traps and their energy dependence were
calculated using UV-assisted CV technique as low as +7.8 $\times$ 1011 cm-2.
Conduction band offset using the linear relationship between flat band voltage
and oxide thicknesses is calculated to be 1.7 eV. Furthermore, the average
breakdown fields for the MOCVD grown Al2O3 material is found to be
approximately 5.8 MV/cm. This approach of in-situ dielectric deposition on
$\beta$-Ga2O3 can pave the way as gate dielectrics for future $\beta$-Ga2O3
based high performance MOSFETs due to its promise of improved interface and
bulk quality compared to other conventional dielectric deposition techniques.

## Experimental Section

The MOSCAP fabrication started with the growth of a 500 nm $\beta$-Ga2O3
epilayer on Sn-doped bulk (010) $\beta$-Ga2O3 substrates (NCT Japan) using
Agnitron MOVPE reactor with far injection showerhead design. Prior to loading
the sample in the growth chamber, the substrates were cleaned using acetone,
methanol and DI water and followed by a 15 min dip in HF solution.
Triethylgallium (TEGa) and O2 were used as the precursors, Ar as the carrier
gas and diluted silane as the dopant. The growth process of $\beta$-Ga2O3 was
performed at a low temperature of 600 0C and at a chamber pressure of 60
Torr.[35, 36] The gas flow values for TEGa and O2 were set to 65 and 800 sccm
and the Ar flow rate was set to 1100 sccm. A silane flow rate of 0.06 nmol/min
was used to achieve a silicon doping concentration of 1017 cm-3. After the
Ga2O3 growth, a 30 second purge step was used to purge all the unreacted
precursors from the reactor. During this purge step, the reactor pressure was
ramped down from 60 to 15 Torr whereas the temperature is kept same at 600 0C.
The O2 flow rate was also reduced from 800 to 500 sccm during this 30 second
purge step. Following this, growth time of the Al2O3 layer was varied to
achieve dielectric thicknesses of 14, 24 and 55 nm, on three different
substrates. In-situ Al2O3 was grown at a growth rate of 1.2 nm/min using the
Trimethylaluminum (TMAl) precursor with a flow rate of 5.33 $\mu$mol/min and
O2 with a flow rate of 500 sccm. To find the thickness of the dielectric Al2O3
layer, the samples were patterned using photolithography and the Al2O3
dielectric layer was etched away using buffered oxide etch (BOE) with
photoresist as the mask. The etch depth is measured using atomic force
microscopy and the details are provided in the supporting information. After
the growth of the Al2O3 layer, Ti (50 nm)/Au (100nm) ohmic contacts were
sputter deposited on the back side of the MOSCAPs. Finally, circular Ni (10
nm)/Au (50 nm) gate contacts were patterned using standard photolithography
and deposited on the Al2O3 surface using E-beam evaporation.

Supporting Information

Supporting Information is available from the author.

Acknowledgements

This material is based upon work supported by the Air Force Office of
Scientific Research under Award Number FA9550-18-1-0507 (Program Manager: Dr.
Ali Sayir). Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the United States Air Force. The authors also acknowledge
funding from the National Science Foundation (NSF) through Grant DMR-1931652.
The work at Penn State University was supported by the Air Force Office of
Scientific Research (AFOSR) program FA9550-18-1-0277 and GAME MURI,
10059059-PENN.

## References

  * [1] S. Pearton, J. Yang, P. H. Cary IV, F. Ren, J. Kim, M. J. Tadjer, M. A. Mastro,  _Applied Physics Reviews_ 2018, _5_ , 1 011301.
  * [2] S. Stepanov, V. Nikolaev, V. Bougrov, A. Romanov,  _Rev. Adv. Mater. Sci_ 2016, _44_ 63\.
  * [3] K. Zeng, Y. Jia, U. Singisetti,  _IEEE Electron Device Letters_ 2016, _37_ , 7 906.
  * [4] K. Zeng, U. Singisetti,  _Applied Physics Letters_ 2017, _111_ , 12 122108.
  * [5] K. D. Chabak, J. P. McCandless, N. A. Moser, A. J. Green, K. Mahalingam, A. Crespo, N. Hendricks, B. M. Howe, S. E. Tetlak, K. Leedy, R. C. Fitch, D. Wakimoto, K. Sasaki, A. Kuramata, G. H. Jessen,  _IEEE Electron Device Letters_ 2018, _39_ , 1 67.
  * [6] A. J. Green, K. D. Chabak, M. Baldini, N. Moser, R. Gilbert, R. C. Fitch, G. Wagner, Z. Galazka, J. Mccandless, A. Crespo, K. Leedy, G. H. Jessen,  _IEEE Electron Device Letters_ 2017, _38_ , 6 790.
  * [7] T.-H. Hung, K. Sasaki, A. Kuramata, D. N. Nath, P. Sung Park, C. Polchinski, S. Rajan,  _Applied Physics Letters_ 2014, _104_ , 16 162106.
  * [8] D. Biswas, C. Joishi, J. Biswas, K. Thakar, S. Rajan, S. Lodha,  _Applied Physics Letters_ 2019, _114_ , 21 212106.
  * [9] P. H. Carey IV, J. Yang, F. Ren, R. Sharma, M. Law, S. J. Pearton,  _ECS Journal of Solid State Science and Technology_ 2019, _8_ , 7 Q3221.
  * [10] N. A. Moser, J. P. McCandless, A. Crespo, K. D. Leedy, A. J. Green, E. R. Heller, K. D. Chabak, N. Peixoto, G. H. Jessen,  _Applied Physics Letters_ 2017, _110_ , 14 143505.
  * [11] N. K. Kalarickal, Z. Feng, A. F. M. Anhar Uddin Bhuiyan, Z. Xia, W. Moore, J. F. McGlone, A. R. Arehart, S. A. Ringel, H. Zhao, S. Rajan,  _IEEE Transactions on Electron Devices_ 2021, _68_ , 1 29.
  * [12] Z. Xia, H. Chandrasekar, W. Moore, C. Wang, A. J. Lee, J. McGlone, N. K. Kalarickal, A. Arehart, S. Ringel, F. Yang, et al.,  _Applied Physics Letters_ 2019, _115_ , 25 252104.
  * [13] H. N. Masten, J. D. Phillips, R. L. Peterson,  _IEEE Transactions on Electron Devices_ 2019, _66_ , 6 2489.
  * [14] K. D. Chabak, D. E. Walker, A. J. Green, A. Crespo, M. Lindquist, K. Leedy, S. Tetlak, R. Gilbert, N. A. Moser, G. Jessen,  In _2018 IEEE MTT-S International Microwave Workshop Series on Advanced Materials and Processes for RF and THz Applications (IMWS-AMP)_. 2018 1–3.
  * [15] K. D. Chabak, N. Moser, A. J. Green, D. E. Walker Jr, S. E. Tetlak, E. Heller, A. Crespo, R. Fitch, J. P. McCandless, K. Leedy, et al.,  _Applied Physics Letters_ 2016, _109_ , 21 213501.
  * [16] A. J. Green, K. D. Chabak, E. R. Heller, R. C. Fitch, M. Baldini, A. Fiedler, K. Irmscher, G. Wagner, Z. Galazka, S. E. Tetlak, A. Crespo, K. Leedy, G. H. Jessen,  _IEEE Electron Device Letters_ 2016, _37_ , 7 902.
  * [17] W. Li, K. Nomoto, Z. Hu, T. Nakamura, D. Jena, H. G. Xing,  In _2019 IEEE International Electron Devices Meeting (IEDM)_. 2019 12.4.1–12.4.4.
  * [18] W. Li, K. Nomoto, Z. Hu, D. Jena, H. G. Xing,  _IEEE Electron Device Letters_ 2020, _41_ , 1 107.
  * [19] W. Li, K. Nomoto, Z. Hu, D. Jena, H. G. Xing,  In _2019 Device Research Conference (DRC)_. 2019 209–210.
  * [20] Z. Jian, S. Mohanty, E. Ahmadi,  _Applied Physics Letters_ 2020, _116_ , 24 242105.
  * [21] X. Liu, S. Chan, F. Wu, Y. Li, S. Keller, J. Speck, U. Mishra,  _Journal of crystal growth_ 2014, _408_ 78\.
  * [22] X. Liu, J. Kim, R. Yeluri, S. Lal, H. Li, J. Lu, S. Keller, B. Mazumder, J. Speck, U. Mishra,  _Journal of Applied Physics_ 2013, _114_ , 16 164507\.
  * [23] X. Liu, R. Yeluri, J. Kim, S. Lal, A. Raman, C. Lund, S. Wienecke, J. Lu, M. Laurent, S. Keller, et al.,  _Applied Physics Letters_ 2013, _103_ , 5 053509.
  * [24] X. Liu, J. Kim, D. Suntrup, S. Wienecke, M. Tahhan, R. Yeluri, S. Chan, J. Lu, H. Li, S. Keller, et al.,  _Applied Physics Letters_ 2014, _104_ , 26 263511.
  * [25] C. Gupta, S. H. Chan, A. Agarwal, N. Hatui, S. Keller, U. K. Mishra,  _IEEE Electron Device Letters_ 2017, _38_ , 11 1575\.
  * [26] G. Cliff, G. Lorimer,  _Journal of Microscopy_ 1975, _103_ , 2 203.
  * [27] R. Winter, J. Ahn, P. C. McIntyre, M. Eizenberg,  _Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena_ 2013, _31_ , 3 030604.
  * [28] Z. Zhang, C. M. Jackson, A. R. Arehart, B. McSkimming, J. S. Speck, S. A. Ringel,  _Journal of electronic materials_ 2014, _43_ , 4 828\.
  * [29] B. Swenson, U. Mishra,  _Journal of Applied Physics_ 2009, _106_ , 6 064902\.
  * [30] W. Liu, I. Sayed, C. Gupta, H. Li, S. Keller, U. Mishra,  _Applied Physics Letters_ 2020, _116_ , 2 022104.
  * [31] R. Yeluri, X. Liu, B. L. Swenson, J. Lu, S. Keller, U. K. Mishra,  _Journal of Applied Physics_ 2013, _114_ , 8 083718\.
  * [32] F. Akyol,  _Journal of Applied Physics_ 2020, _127_ , 7 074501\.
  * [33] _Sentaurus Device User Manual, (Version N-2017.09, March 2017)_.
  * [34] S. H. Chan,  Ph.D. thesis, UC Santa Barbara, 2018.
  * [35] A. Bhattacharyya, P. Ranga, S. Roy, J. Ogle, L. Whittaker-Brooks, S. Krishnamoorthy,  _Applied Physics Letters_ 2020, _117_ , 14 142102.
  * [36] P. Ranga, A. Bhattacharyya, A. Chmielewski, S. Roy, N. Alem, S. Krishnamoorthy,  _Applied Physics Letters_ 2020, _117_ , 17 172105.