# Deep Lake: a Lakehouse for Deep Learning

Sasun Hambardzumyan, Abhinav Tuli, Levon Ghukasyan, Fariz Rahman, Hrant
Topchyan, David Isayan, Mikayel Harutyunyan, Tatevik Hakobyan, Ivo Stranic,
Davit Buniatyan<EMAIL_ADDRESS>Activeloop238 Castro streetMountain
ViewCAUSA94043

(2022)

###### Abstract.

Traditional data lakes provide critical data infrastructure for analytical
workloads by enabling time travel, running SQL queries, ingesting data with
ACID transactions, and visualizing petabyte-scale datasets on cloud storage.
They allow organizations to break down data silos, unlock data-driven
decision-making, improve operational efficiency, and reduce costs. However, as
deep learning takes over common analytical workflows, traditional data lakes
become less useful for applications such as natural language processing (NLP),
audio processing, computer vision, and applications involving non-tabular
datasets.

This paper presents Deep Lake, an open-source lakehouse for deep learning
applications developed at Activeloop111Source code available:
https://github.com/activeloopai/deeplake222 Documentation available at
https://docs.deeplake.ai. Deep Lake maintains the benefits of a vanilla data
lake with one key difference: it stores complex data, such as images, videos,
annotations, as well as tabular data, in the form of tensors and rapidly
streams the data over the network to (a) Tensor Query Language, (b) in-browser
visualization engine, or (c) deep learning frameworks without sacrificing GPU
utilization. Datasets stored in Deep Lake can be accessed from PyTorch(Paszke
et al., 2017), TensorFlow(Abadi et al., 2016), JAX(Bradbury et al., 2018), and
integrate with numerous MLOps tools.

Deep Lake, Deep Learning, Data Lake, Lakehouse, Cloud Computing, Distributed
Systems

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference:  ; ;

##  1\. Introduction

A data lake is a central repository that allows organizations to store
structured, unstructured, and semi-structured data in one place. Data lakes
provide a better way to manage, govern, and analyze data. In addition, they
provide a way to break data silos and gain insights previously hidden in
disparate data sources. First-generation data lakes traditionally collected
data into distributed storage systems such as HDFS (Shvachko et al., 2010) or
AWS S3(s3, 2006). Unorganized collections of the data turned data lakes into
”data swamps”, which gave rise to the second generation data lakes led by
Delta, Iceberg, and Hudi (Armbrust et al., 2020; ice, 2018; hud, 2017). They
strictly operate on top of standardized structured formats such as Parquet,
ORC, Avro (Vohra, 2016; orc, 2015; avr, 2019) and provide features like time
travel, ACID transactions, and schema evolution. Data lakes directly integrate
with query engines such as Presto, Athena, Hive, Photon (Sethi et al., 2019;
ath, 2018; Thusoo et al., 2010; Samwel et al., 2022) to run analytical
queries. Additionally, they connect to frameworks like Hadoop, Spark, and
Airflow (had, 2018; Zaharia et al., 2010; air, 2017) for ETL pipelines
maintenance. In its turn, the integration between data lakes and query engines
with clear compute and storage separation resulted in the emergence of systems
like Lakehouse (Armbrust et al., 2021) that serve as an alternative to data
warehouses, including Snowflake, BigQuery, Redshift, and Clickhouse(Dageville
et al., 2016; big, 2012; Gupta et al., 2015; cli, 2009).

Over the past decade, deep learning has outpaced traditional machine learning
techniques involving unstructured and complex data such as text, images,
videos, and audio (Krizhevsky et al., 2012; LeCun et al., 2015; Goodfellow et
al., 2016; Zhang et al., 2015; Mikolov et al., 2013; Bahdanau et al., 2014;
Redmon et al., 2016; Oord et al., 2016). Not only did deep learning systems
outgrow traditional techniques, but they also achieved super-human accuracy in
applications such as cancer detection from X-Ray images, anatomical
reconstruction of human neural cells, playing games, driving cars, unfolding
proteins, and generating images (Rajpurkar et al., 2017; Lee et al., 2017;
Silver et al., 2018; Huang and Chen, 2020; Tunyasuvunakool et al., 2021).
Large language models with transformer based architectures achieved state-of-
the-art results across translation, reasoning, summarization, and text
completion tasks (Vaswani et al., 2017; Devlin et al., 2018; Yang et al.,
2019; Brown et al., 2020). Large multi-modal networks embed unstructured data
into vectors for cross-modal search (Baevski et al., 2022; Radford et al.,
2021). Moreover, they are used to generate photo-realistic images from text
(Ramesh et al., 2021; Saharia et al., 2022). Although one of the primary
contributors to the success of deep learning models has been the availability
of large datasets such as CoCo (330K images), ImageNet (1.2M images), Oscar
(multilingual text corpus), and LAION (400M and 5B images) (Lin et al., 2014;
Deng et al., 2009a; Suárez et al., 2019; Schuhmann et al., 2021), it does not
have a well-established data infrastructure blueprint similar to traditional
analytical workloads to support such scale. On the other hand, Modern Data
Stack (MDS) lacks features required to deploy performant deep learning-based
solutions, so organizations opt to develop in-house systems.

Figure 1. Machine Learning Loop with Deep Lake

In this paper, we introduce Deep Lake, a lakehouse specialized for deep
learning workloads. Deep Lake retains the benefits of a traditional data lake
with one notable distinction: it stores complex data, such as images, videos,
annotations, and tabular data, as tensors and rapidly streams the data to deep
learning frameworks over the network without sacrificing GPU utilization.
Furthermore, it provides native interoperability between deep learning
frameworks such as PyTorch, TensorFlow, and JAX (Paszke et al., 2017; Abadi et
al., 2016; Bradbury et al., 2018).

The remainder of this paper unfolds as follows. We begin by considering
current challenges in deep learning on unstructured data. Next, we present the
Deep Lake storage format with its key concepts. Furthermore, we discuss Deep
Lake capabilities and its applications within the ML cycle. Next, we provide
performance experiments and discuss the results. Finally, we review related
work, list possible limitations, and conclude.

##  2\. Current Challenges

In this section, we discuss the current and historical challenges of
unstructured or complex data management.

###  2.1. Complex Data Types in a Databases

There are three widely accepted reasons why blobs or binary data such as
images should not be stored in a database. Firstly, databases typically
operate on row-wise storage architecture. Therefore, running a query becomes
significantly slower if a row contains large amounts of data. Secondly,
persisting data in hot memory is expensive compared to cold storage. Lastly,
storing images in a RAM can quickly outgrow in-memory capacity and potentially
cause a database to fail. In effect, this renders most widely-used data
infrastructure tools unsuitable for processing unstructured and complex data
types - due to speed, cost, and reliability concerns.

###  2.2. Complex Data Along with Tabular Formats

Increases in large-scale analytical and BI workloads motivated the development
of compressed structured formats like Parquet, ORC, Avro, or transient in-
memory formats like Arrow(Vohra, 2016; orc, 2015; avr, 2019; arr, 2018). As
tabular formats gained adoption, attempts to extend those formats, such as
Petastorm (Pet, 2018) or Feather (fea, 2016)for deep learning, have emerged.
To the best of our knowledge, these formats have yet to gain wide adoption.
This approach primarily benefits from native integrations with Modern Data
Stack (MDS). However, as discussed previously, upstream tools require
fundamental modifications to adapt to deep learning applications.

###  2.3. Object Storage for Deep Learning

The current cloud-native choice for storing large unstructured datasets is
object storage such as AWS S3(s3, 2006), Google Cloud Storage (GCS)(gcs,
2010)or MinIO(min, 2018). Object storage does offer three main benefits over
distributed network file systems. They are (a) cost-efficient, (b) scalable,
and (c) serve as a format-agnostic repository. However, cloud storages are not
without drawbacks. Firstly, they introduce significant latency overhead,
especially when iterating over many small files such as text or JSON. Next,
unstructured data ingestion without metadata control can produce ”data
swamps”. Furthermore, object storage has built-in version control; it is
rarely used in data science workflows. Lastly, data on object storage gets
copied to a virtual machine before training, thus resulting in storage
overhead and additional costs.

###  2.4. Second Generation of Data Lakes

The second generation data lakes led by Delta, Iceberg, Hudi (Armbrust et al.,
2020; ice, 2018; hud, 2017)overcome limitations of object storage by managing
tabular format files with the following primary properties.

  1. (1)

Update operations: inserting or deleting a row on top of a tabular format
file.

  2. (2)

Streaming: downstream data ingestion with ACID properties and upstream
integration with query engine exposing SQL interface.

  3. (3)

Schema evolution: evolving columnar structure while preserving backward
compatibility.

  4. (4)

Time travel and audit log trailing: preserving historical state with rollback
property where queries can be reproducible. Also, support for row-level
control on data lineage.

  5. (5)

Layout optimization: Built-in feature to optimize file sizes and data
compaction with custom ordering support. Significantly speeds up querying.

However, second-generation data lakes are still bound by the limitations of
the inherent data formats to be used in deep learning, as previously discussed
in section 2.2. Hence in this paper, we extend the second generation of data
lake capabilities for deep learning use cases by rethinking the format and
upstream features, including querying, visualization, and native integration
to deep learning frameworks.

##  3\. Deep Lake Storage Format

Deep Lake datasets follow columnar storage architecture, with tensors as
columns, as shown in Fig. 2. Each tensor is a collection of _chunks_ \- binary
blobs that contain the data samples. An index map associated with each tensor
helps find the right chunk and index of the sample within that chunk for a
given sample index.

###  3.1. Dataset

A sample in a dataset represents a single row indexed across parallel tensors.
As opposed to a document storage format, sample elements are logically
independent, which enables partial access to samples for running performant
queries or streaming selected tensors over the network to the GPU training
instances. Multiple tensors can be grouped. Groups implement syntactic nesting
and define how tensors are related to each other. Syntactic nesting avoids the
format complication for hierarchical memory layout. Changes to the dataset’s
schema are also tracked over time with version control, similar to dataset
content changes.

###  3.2. Tensors

Tensors are typed and can be appended or in-place modified. Default access to
an index or a set of indices returns the data as NumPy arrays(Oliphant, 2006).
Instead of storing 1-D data as seen in Parquet (Vohra, 2016) or series in
Arrow(arr, 2018), tensors can accommodate n-dimensional data, where typically
the first dimension corresponds to the index or batch dimension. Tensors can
contain dynamically shaped arrays, also called ragged tensors, as opposed to
other statically chunked array formats such as Zarr (Miles et al., 2020).

###  3.3. Types

Htype defines the expectations on samples in a tensor such as data type (dtype
as seen in NumPy(Oliphant, 2006)), shape, number of dimensions, or
compression. Typed tensors make interacting with deep learning frameworks
straightforward and enable sanity checks and efficient memory layout. By
inheriting from a generic tensor htype, we can construct types such as image,
video, audio, bbox, dicom, and others. For example, a tensor with image htype
would expect samples being appended to it to have dtype as uint8 and shape
length 3 (i.e. width, height and number of channels). We further expand on the
notion of htypes allowing for meta types that support storing image sequences
in tensors (sequence[image]), referencing to remotely stored images, while
maintaining the regular behavior of a image tensor (link[image]), or even
possible cross-format support.

###  3.4. Memory Layout

Figure 2. How each sample (row) is stored in a set of columnar tensors with
dynamically sized chunks

Tensors are stored in chunks at the storage level. While statically (inferred)
shaped chunking avoids maintaining a chunk map table, it introduces
significant user overhead during the specification of the tensor, custom
compression usage limitations, underutilized storage for dynamically shaped
tensors, and post-processing inefficiencies. Deep Lake chunks are constructed
based on the lower and upper bound of the chunk size to fit a limited number
of samples. This comes with a trade-off of having a compressed index map that
preserves the sample index to chunk id mapping per tensor while enabling chunk
sizes in the range optimal for streaming while accommodating mixed shape
samples. One could consider the approach taken in this paper as an optimized
trade-off between file system page map and compute-defined map-less array
storage system. For practical reasons, a single chunk encoder can be scaled to
billions of images while maintaining a 150MB chunk encoder per 1PB tensor
data. Further scaling can be introduced by sharding the chunk encoder. Chunks
contain header information such as byte ranges, shapes of the samples, and the
sample data itself. If a sample is larger than the upper bound chunk size,
which is the case for large aerial or microscopy images, the sample is tiled
into chunks across spatial dimensions. The only exception to tiling is videos.
Videos are preserved due to efficient frame mapping to indices, only key-frame
decompression, and range-based requests while streaming.

###  3.5. Access Patterns

Deep Lake storage format is optimized for deep learning training and
inference, including sequential and random access. Sequential access is used
for running scan queries, transforming tensors into other tensors, or running
inference. Random access use cases include multiple annotators writing labels
to the same image or models storing back predictions along with the dataset.
While the strict mode is disabled, out-of-the-bounds indices of a tensor can
be assigned, thus accommodating sparse tensors. However, random assignment
over time will produce inefficiently stored data chunks. To fix the data
layout, we implement an on-the-fly re-chunking algorithm to optimize the data
layout. One of the key access patterns of Deep Lake is shuffled stream access
for training machine learning models. It requires random or custom order
access while streaming chunks into the training process. This is achieved by
involving range-based requests to access sub-elements inside chunks, running
complex queries before training to determine the order, and maintaining a
buffer cache of fetched and unutilized data. This avoids having a separate
compute cluster for running shuffling algorithm (Luan et al., 2022).

###  3.6. Storage Providers

Deep Lake can be plugged into any storage provider, including object storages
such as AWS S3(s3, 2006), Google Cloud Storage (GCS)(gcs, 2010), POSIX
compatible file systems or local in-memory storage. Moreover, it constructs
memory caching by chaining various storage providers together, for instance -
Least Recently Used (LRU) cache of remote S3 storage with local in-memory
data.

##  4\. Deep Lake System Overview

###  4.1. Version Control

Figure 3. Version History of Evolving Deep Lake Dataset

Deep Lake also addresses the need for reproducibility of experiments and
compliance, with a complete data lineage. Different versions of the dataset
exist in the same storage, separated by sub-directories. Each sub-directory
acts as an independent dataset with its individual metadata files. Unlike a
non-versioned dataset, these sub-directories only contain chunks modified in
the particular version, along with a corresponding chunk_set per tensor
containing the names of all the modified chunks. A version control info file
present at the root of the directory keeps track of the relationship between
these versions as a branching version-control tree. While accessing any chunk
of a tensor at a particular version, the version control tree is traversed
starting from the current commit, heading towards the first commit. During the
traversal, the chunk set of each version is checked for the existence of the
required chunk. If the chunk is found, the traversal is stopped, and data is
retrieved. For keeping track of differences across versions, for each version,
a commit diff file is also stored per tensor. This makes it faster to compare
across versions and branches. Moreover, ids of samples are generated and
stored during dataset population. This is important for keeping track of the
same samples during merge operations. Deep Lake’s version control interface is
the Python API, which enables machine learning engineers to version their
datasets within their data processing scripts without switching back and forth
from the CLI. It supports the following commands:

  * •

Commit: creates an immutable snapshot of the current state of the dataset.

  * •

Checkout: checks out to an existing branch/commit or creates a new branch if
one doesn’t exist.

  * •

Diff: compares the differences between 2 versions of the dataset.

  * •

Merge: merges two different versions of the dataset, resolving conflicts
according to the policy defined by the user.

###  4.2. Visualization of Tensors

Data visualization is a crucial part of ML workflows, especially when the data
is hard to parse analytically. Fast and seamless visualization allows faster
data collection, annotation, quality inspection, and training iterations. The
Deep Lake visualizer engine provides a web interface for visualizing large-
scale data directly from the source. It considers htype of the tensors to
determine the best layout for visualization. Primary tensors, such as image,
video and audio are displayed first, while secondary data and annotations,
such as text, class_label, bbox and binary_mask are overlayed. The visualizer
also considers the meta type information, such as sequence to provide a
sequential view of the data, where sequences can be played and jump to the
specific position of the sequence without fetching the whole data, which is
relevant for video or audio use cases. Visualizer addresses critical needs in
ML workflows, enabling users to understand and troubleshoot the data, depict
its evolution, compare predictions to ground truth, or display multiple
sequences of images (e.g. camera images and disparity maps) side-by-side.

###  4.3. Tensor Query Language

Querying and balancing datasets is a common step in training deep learning
workflows. Typically, this is achieved inside a dataloader using sampling
strategies or separate pre-processing step to sub-select the dataset. On the
other hand, traditional data lakes connect to external analytical query
engines (Samwel et al., 2022) and stream Dataframes to data science workflows.
To resolve the gap between the format and fast access to the specific data, we
provide an embedded SQL-like query engine implemented in C++ called Tensor
Query Language (TQL). We parse SQL query into a computational graph of tensor
operations, provide the query plan to the scheduler and execute the query.
Execution of the query can be delegated to external tensor computation
frameworks such as PyTorch (Paszke et al., 2017) or XLA (Sabne, 2020) and
efficiently utilization underlying accelerated hardware. In addition to
standard SQL features, TQL also implements numeric computation. There are two
main reasons for implementing a new query language. First, traditional SQL
does not support multidimensional array operations such as computing mean of
the image pixels or projecting arrays on a specific dimension. TQL solves this
by adding Python/NumPy style indexing, slicing of arrays, and providing large
set of convenience functions to work with arrays, many of which are common
operations supported in NumPy. Second, TQL enables a deeper integration of the
query with other features of the Deep Lake, such as version control, streaming
engine, and visualization. For example, TQL allows querying data on the
specific versions or potentially across multiple versions of a dataset. TQL
also supports specific instructions to customize visualization of the query
result or seamless integration with the dataloader for filtered streaming.
Embedded query engine runs along with the client either while training a model
on a remote compute instance or in-browser compiled over WebAssembly. TQL
extends SQL with numeric computations on top of multi-dimensional columns. It
constructs views of datasets, which can be visualized or directly streamed to
deep learning frameworks. Query views, however can be sparse, which can affect
streaming performance.

    SELECT
      images[100:500, 100:500, 0:2] as crop,
      NORMALIZE(
        boxes,
        [100, 100, 400, 400]) as box
    FROM
      dataset
    WHERE IOU(boxes, "training/boxes") > 0.95
    ORDER BY IOU(boxes, "training/boxes")
    ARRANGE BY labels

Figure 4. An example query that arranges cropped images ordered by bounding
boxes predictions error measured over user-defined function IOU (Intersection
over Union).

###  4.4. Materialization

Most of the raw data used for deep learning is stored as raw files (compressed
in formats like JPEG), either locally or on the cloud. A common way to
construct datasets is to preserve pointers to these raw files in a database,
query this to get the required subset of data, fetch the filtered files to a
machine, and then train a model iterating over files. In addition, data
lineage needs to be manually maintained with a provenance file. Deep Lake
format simplifies these steps using linked tensors - storing pointers
(links/urls to one or multiple cloud providers) to the original data. The
pointers within a single tensor can be connected to multiple storage
providers, thus allowing users to get a consolidated view of their data
present in multiple sources. All of Deep Lake’s features including queries,
version control and streaming to deep learning frameworks can be used with
linked tensors. However, the performance of data streaming will not be as
optimal as default tensors. A similar problem exists with sparse views created
due to queries, which would be inefficiently streamed due to the chunk
layout.Furthermore, materialization transforms the dataset view into an
optimal layout to stream into deep learning frameworks to iterate faster.
Materialization involves fetching the actual data from links or views, and
efficiently laying out into chunks. Performing this step towards the end of
machine learning workflows leads to minimum data duplication, while ensuring
ensures optimal streaming performance and minimal data duplication, with full
data lineage.

###  4.5. Streaming Dataloader

As datasets become larger, storing and transferring over network from a remote
distributed storage becomes inevitable. Data streaming enables training models
without waiting for all of the data to be copied to a local machine. The
streaming dataloader ensures data fetching, decompression, applying
transformations, collation, and data handover to the training model. Deep
learning data-loaders typically delegate fetching and transformation to
parallel running processes to avoid synchronous computation. Then the data is
transferred to the main worker through inter-process communication (IPC) which
introduces memory copy overhead or using shared memory with some reliability
issues. In contrast, Deep Lake dataloader delegates highly parallel fetching
and in-place decompressing in C++ per process to avoid global interpreter
lock. Then, it passes the in-memory pointer to user-defined transformation
function and collates them before exposing to the training loop in deep
learning native memory layout. Transformation concurrently executes in
parallel when it uses only native library routines calls and release python
global interpreter lock (GIL) accordingly. As a result, we get:

  * •

Performance: Delivering data to the deep learning model fast enough so that
either the GPU is fully utilized or bottlenecked by the compute.

  * •

Smart Scheduler: Dynamically differentiating between CPU-intensive jobs
prioritization over less-intensive.

  * •

Efficient Resource Allocation: Predicting memory consumption to avoid breaking
the training process due to memory overfilling.

Figure 5. Conversion of raw CIFAR (30x30) images on the filesystem to the
corresponding data formats where Deep Lake is comparable to Petastorm (Parquet
extension). b) iterate over all the samples in CIFAR (30x30) once stored in
the corresponding format c) local iteration of the random images dataset d)
remote dataloader iteration of random image dataset

##  5\. Use Cases

In this section, we review the applications of Deep Lake in various use cases.

###  5.1. Machine Learning Use Cases

####  5.1.1. Deep Learning Model Training

Deep learning models are trained at multiple levels in an organization,
ranging from exploratory training occurring on personal computers to large-
scale training that occurring on distributed machines involving many GPUs. The
time and effort required to bring the data from long-term storage to the
training client is often comparable to the training itself. Deep Lake solves
this problem by enabling rapid streaming of data without bottlenecking the
downstream training process, thus avoiding the cost and time required to
duplicate data on local storage.

####  5.1.2. Data Lineage and Version Control

Deep learning data constantly evolves as new data is added and existing data
is quality controlled. Analytical and training workloads occur in parallel
while the data is changing. Hence, knowing which data version was used by a
given workload is critical to understand the relationship between the data and
model performance. Deep Lake enables deep learning practitioners to understand
which version of their data was used in any analytical workload and to time
travel across these versions if an audit is required. Since all data is
mutable, it can be edited to meet compliance-related privacy requirements.
Like Git for code, Deep Lake also introduces the concept of data branches,
allowing experimentation and editing of data without affecting colleagues’
work.

####  5.1.3. Data Querying and Analytics

Training of deep learning models rarely occurs on all data collected by an
organization for a particular application. Training datasets are often
constructed by filtering the raw data based on conditions increasing model
performance, which often includes data balancing, eliminating redundant data,
or selecting data that contains specific features. Deep Lake provides the
tools to query and analyze data so that deep learning engineers can create
datasets yielding the highest accuracy models.

####  5.1.4. Data Inspection and Quality Control

Though unsupervised learning is becoming more applicable in real-world use
cases, most deep learning applications still rely on supervised learning. Any
supervised learning system is only as good as the quality of its data, often
achieved by manual and exhaustive inspection of the data. Since this process
is time-consuming, it is critical to provide the humans in the loop with tools
to examine vast amounts of data very quickly. Deep Lake allows to inspect deep
learning datasets of any size from the browser without any setup time or need
to download data. Furthermore, the tools can be extended for comparing model
results with ground truth. Combined with querying and version control, this
can be applied to the iterative improvement of data to achieve the best
possible model.

##  6\. Performance Benchmarks

In this section, we experimentally demonstrate Deep Lake’s state-of-the-art
performance at scale, against other dataloaders, and contrast using Deep Lake
to an object storage.

###  6.1. Comparison with Other Dataloaders

All experiments shown in Fig. 5 were carried out on p3.2xlarge EC2 instance on
AWS with one V100 GPU card. The datasets used were the CIFAR-10 dataset and
the random dataset. CIFAR-10 consists of colored images with size (30x30)
stored in raw JPEG file with the labels stored in the corresponding TXT files.
The random dataset is a generated dataset consisting of colored (250x250)
images stored as JPEG files. The number of images in both the datasets were
50,000. The list of libraries in which the benchmarks were carried out were
Hub, FFCV (Guillaume Leclerc, 2021), Squirrel (Team, 2022), Webdataset (web,
2018) and Petastorm (Pet, 2018). For more detailed dataloader benchmarks, we
would recommend an exhaustive datalaoder overview study by Ofeidis et al.
(forthcoming) (Iason Ofeidis, 2022).

###  6.2. Streamable Training on the Cloud

####  6.2.1. Single GPU training on ImageNet compared to cloud storage

Figure 6. Training on ImageNet on an S3: AWS File Mode copies file by file
from S3; Fast File Mode starts immediately with slower training; Deep Lake
performs as if data is local, although it is streamed

We take ImageNet dataset (Deng et al., 2009b) and store it on AWS S3 (s3,
2006) as original and Deep Lake format. The dataset contains 1.2 million
images and labels in total 150GB. Deep Lake achieves virtually similar
training performance as if the data was local to the machine. This saves up to
4x GPU compute time and cost as shown in Fig. 6

####  6.2.2. Distributed training of a large multi-modal dataset

Figure 7. GPU Utilization of 16xA100 GPUs on the same machine while training
1B parameter CLIP model on LAION-400M streaming the dataset from us-east to
us-central.

As a second experiment, we take LAION dataset (Schuhmann et al., 2022)
containing 400M image-text pairs and train CLIP (Radford et al., 2021), image-
text embedding model with 1 billion parameters. The original dataset is a
table of Parquet files with a column of image URLs. Fetching the data using
the LAION toolkit takes 100 hours and only 6 hours to ingest into Deep Lake
dataset on S3. The dataset download from the source took 100 hours, while
ingestion to Deep Lake format took only 6 hours, totaling 1.9TB in size. The
dataset has been stored on AWS in the US-east region while training GPU
machine in the US-central region. As shown on Fig. 7 Deep Lake achieves high
GPU utilization by streaming 5,100 images/s into 16 Nvidia A100 GPUs while
without model up to 80,000 images/s per machine on the same region.

##  7\. Discussion and Limitations

Deep Lake’s primary use cases include (a) Deep Learning Model Training, (b)
Data Lineage and Version Control, (c) Data Querying, and Analytics, (d) Data
Inspection and Quality Control. Deep Lake achieves state-of-the-art results in
local and remote settings, as seen in benchmarks for iterating on large
images. Primarily, it has been faster than FFCV (Guillaume Leclerc, 2021),
which claimed a reduction of ImageNet model training up to 98 cents per model
training. Furthermore, Deep Lake achieves similar ingestion performance to
Petastorm (Pet, 2018) (referred to extended Parquet). Although the iteration
on smaller images is faster on Petastorm, Deep Lake significantly outperforms
on larger images. Parquet is optimized for small cells and analytical
workloads, while Deep Lake is optimized for large, dynamically shaped
tensorial data. Compared to other data lake solutions, its minimal python
package design enables Deep Lake to be easily integrated into large-scale
distributed training or inference workloads. The current implementation of
Deep Lake has opportunities for further improvement. Firstly, the storage
format does not support custom ordering for an even more efficient storage
layout required for vector search or key-value indexing. Secondly, Deep Lake
implements branch-based locks for concurrent access. Similar to Delta ACID
transaction model (Armbrust et al., 2020), Deep Lake can be extended to
highly-performant parallel workloads. Thirdly, the current implementation of
TQL only supports a subset of SQL operations (i.e., does not support
operations such as join). Further work will focus on making it SQL-complete,
extending to more numeric operations, and benchmarking against SQL engines.

##  8\. Related Work

Deep Lake builds on numerous prominent contributions across Academia and
industry. For example, Tensor Query Language, similar to TQP (He et al., 2022)
and Velox (Pedro Pedreira, 2022) approaches, runs n-dimensional numeric
operations on tensor storage by truly leveraging the full capabilities of deep
learning frameworks. Multiple projects have tried to improve upon or create
new formats for storing unstructured datasets including TFRecord extending
Protobuf (Pro, 2014) , Petastorm (Pet, 2018) extending Parquet (Vohra, 2016),
Feather(fea, 2016) extending arrow(arr, 2018) , Squirrel using MessagePack
(Team, 2022), Beton in FFCV (Guillaume Leclerc, 2021). Designing a universal
dataset format that solves all use cases is very challenging. Our approach was
mostly inspired by CloudVolume(clo, 2017), a 4-D chunked NumPy storage for
storing large volumetric biomedical data. There are other similar chunked
NumPy array storage formats such as Zarr (Miles et al., 2020),
TensorStore(ten, 2020), TileDB(Papadopoulos et al., 2016). Furthermore, Deep
Lake introduced a typing system, dynamically shaped tensors, integration with
fast deep learning streaming data loaders, and in-browser visualization
support. An alternative approach to store large-scale datasets is to use HPC
distributed file system such as Lustre (Schwan et al., 2003), extending with
PyTorch cache (Kumar and Sivathanu, 2020) or performant storage layer such as
AIStore(Aizman et al., 2019). Deep Lake datasets can be stored on top of POSIX
or REST API-compatible distributed storage systems by leveraging their
benefits. Other comparable approaches evolve in vector databases (Wang et al.,
2021; wea, 2016; Wang et al., 2021) for storing embeddings, feature stores
(Stumpf et al., 2018; fea, 2018) or data version control systems such as DVC
(Kuprieiev et al., 2022), or LakeFS(lak, 2019). In contrast, Deep Lake version
control is in-built into the format without an external dependency, including
Git. Overall, Deep Lake takes an example from data lakes such as Hudi,
Iceberg, Delta (Armbrust et al., 2020; ice, 2018; hud, 2017) and complements
Databarick’s Lakehouse (Armbrust et al., 2021) for Deep Learning applications.

##  9\. Conclusion

We presented Deep Lake, the lakehouse for deep learning. Deep Lake is designed
to help deep learning workflows run as seamlessly as analytical workflows run
on MDS. Notably, Deep Lake is built to retain prominent features of data
lakes, such as time travel, querying, and rapid data ingestion at scale. One
important distinction from traditional data lakes is Deep Lake’s ability to
store unstructured data with all its metadata in deep learning-native columnar
format, which enables rapid data streaming. This allows materializing data
subsets on-the-fly, visualizing them in-browser, or ingesting them into deep
learning frameworks without sacrificing GPU utilization. Finally, we show that
Deep Lake achieves state-of-the-art performance for deep learning on large
datasets via multiple benchmarks.

##  10\. Acknowledgement

The authors would like to thank Richard Socher, Travis Oliphant, Charu
Rudrakshi, Artem Harutyunyan, Iason Ofeidis, Diego Kiedanski, Vishnu Nair,
Fayaz Rahman, Dyllan McCreary, Eduard Grigoryan, Kristina Grigoryan, Ben
Chislett, Joubin Houshyar, Andrii Liubimov, Assaf Pinhasi, Vishnu Nair, Eshan
Arora, Shashank Agarwal, Kristina Arezina, Gevorg Karapetyan, Vigen Sahakyan
and the open-source community including contributors. The project was funded
by Activeloop.

## References

  * (1)
  * s3 (2006) 2006\.  Amazon S3.  _GitHub_ 2022, 1 (2006).  https://aws.amazon.com/s3
  * cli (2009) 2009\.  Clickhouse.  _GitHub_ 2022, 1 (2009).  https://github.com/ClickHouse/ClickHouse
  * gcs (2010) 2010\.  Google Cloud Storage.  _GitHub_ 2022, 1 (2010).  https://cloud.google.com/storage
  * big (2012) 2012\.  Google BigQuery.  _GitHub_ 2022, 1 (2012).  https://cloud.google.com/bigquery
  * Pro (2014) 2014\.  Protocol Buffers - Google’s data interchange format.  _GitHub_ 2022, 1 (2014).  https://github.com/protocolbuffers/protobuf
  * orc (2015) 2015\.  The Apache Software Foundation: Apache ORC.  _GitHub_ 2022, 1 (2015).  https://github.com/apache/orc
  * fea (2016) 2016\.  Feather.  _GitHub_ 2022, 1 (2016).  https://github.com/wesm/feather
  * wea (2016) 2016\.  Weaviate: The ML-first vector search engine.  _GitHub_ 2022, 1 (2016).  https://github.com/semi-technologies/weaviate
  * air (2017) 2017\.  Apache Airflow.  _GitHub_ 2022, 1 (2017).  http://airflow.incubator.apache.org
  * hud (2017) 2017\.  The Apache Software Foundation: Apache Hudi.  _GitHub_ 2022, 1 (2017).  https://hudi.apache.org
  * clo (2017) 2017\.  CloudVolume: IO for Neuroglancer Datasets.  _GitHub_ 2022, 1 (2017).  https://github.com/seung-lab/cloud-volume
  * ath (2018) 2018\.  Amazon Athena.  _GitHub_ 2022, 1 (2018).  https://aws.amazon.com/athena
  * arr (2018) 2018\.  The Apache Software Foundation: Apache Arrow.  _GitHub_ 2022, 1 (2018).  https://arrow.apache.org
  * had (2018) 2018\.  The Apache Software Foundation: Apache Hadoop.  _GitHub_ 2022, 1 (2018).  https://hadoop.apache.org
  * ice (2018) 2018\.  The Apache Software Foundation: Apache Iceberg.  _GitHub_ 2022, 1 (2018).  https://iceberg.apache.org
  * fea (2018) 2018\.  Feast: open source feature store for machine learning.  _GitHub_ 2022, 1 (2018).  https://github.com/feast-dev/feast
  * min (2018) 2018\.  MinIO high performance object storage server compatible with Amazon S3 API.  _GitHub_ 2022, 1 (2018).  https://github.com/minio/minio
  * Pet (2018) 2018\.  Petastorm.  _GitHub_ 2022, 1 (2018).  https://github.com/uber/petastorm
  * web (2018) 2018\.  The WebDataset Format.  _GitHub_ 2022, 1 (2018).  https://github.com/webdataset/webdataset
  * avr (2019) 2019\.  The Apache Software Foundation: Apache Avro.  _GitHub_ 2019, 1 (2019).  https://avro.apache.org
  * lak (2019) 2019\.  LakeFS: data lake with Git-like repository.  _GitHub_ 2022, 1 (2019).  https://github.com/treeverse/lakeFS
  * ten (2020) 2020\.  TensorStore: Library for reading and writing large multi-dimensional arrays.  _GitHub_ 2022, 1 (2020).  https://github.com/google/tensorstore
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: A system for large-scale machine learning. In _12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16)_. 265–283.
  * Aizman et al. (2019) Alex Aizman, Gavin Maltby, and Thomas Breuel. 2019\.  High performance I/O for large scale deep learning. In _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 5965–5967.
  * Armbrust et al. (2020) Michael Armbrust, Tathagata Das, Liwen Sun, Burak Yavuz, Shixiong Zhu, Mukul Murthy, Joseph Torres, Herman van Hovell, Adrian Ionescu, Alicja Łuszczak, et al. 2020\.  Delta lake: high-performance ACID table storage over cloud object stores.  _Proceedings of the VLDB Endowment_ 13, 12 (2020), 3411–3424.
  * Armbrust et al. (2021) Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia. 2021.  Lakehouse: a new generation of open platforms that unify data warehousing and advanced analytics. In _Proceedings of CIDR_.
  * Baevski et al. (2022) Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022\.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  _arXiv preprint arXiv:2202.03555_ (2022).
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Bradbury et al. (2018) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018\.  _JAX: composable transformations of Python+NumPy programs_.  http://github.com/google/jax
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Dageville et al. (2016) Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, et al. 2016\.  The snowflake elastic data warehouse. In _Proceedings of the 2016 International Conference on Management of Data_. 215–226.
  * Deng et al. (2009a) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009a.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Deng et al. (2009b) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009b.  ImageNet: A Large-Scale Hierarchical Image Database. In _CVPR09_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep learning_.  MIT press.
  * Guillaume Leclerc (2021) Andrew Ilyas Sam Park Hadi Salman Guillaume Leclerc, Logan Engstrom. 2021.  The WebDataset Format.  _GitHub_ 2022, 1 (2021).  https://github.com/libffcv/ffcv
  * Gupta et al. (2015) Anurag Gupta, Deepak Agarwal, Derek Tan, Jakub Kulesza, Rahul Pathak, Stefano Stefani, and Vidhya Srinivasan. 2015.  Amazon redshift and the case for simpler data warehouses. In _Proceedings of the 2015 ACM SIGMOD international conference on management of data_. 1917–1923.
  * He et al. (2022) Dong He, Supun Nakandala, Dalitso Banda, Rathijit Sen, Karla Saur, Kwanghyun Park, Carlo Curino, Jesús Camacho-Rodríguez, Konstantinos Karanasos, and Matteo Interlandi. 2022.  Query Processing on Tensor Computation Runtimes.  _arXiv preprint arXiv:2203.01877_ (2022).
  * Huang and Chen (2020) Yu Huang and Yue Chen. 2020\.  Survey of state-of-art autonomous driving technologies with deep learning. In _2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C)_. IEEE, 221–228.
  * Iason Ofeidis (2022) Leandros Tassiulas Iason Ofeidis, Diego Kiedanski. 2022.  An Overview of the Data-loader Landscape: Challanges and Promises.  _unpublished_ (2022).
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.
  * Kumar and Sivathanu (2020) Abhishek Vijaya Kumar and Muthian Sivathanu. 2020.  Quiver: An informed storage cache for deep learning. In _18th USENIX Conference on File and Storage Technologies (FAST 20)_. 283–296.
  * Kuprieiev et al. (2022) Ruslan Kuprieiev, skshetry, Dmitry Petrov, Paweł Redzyński, Peter Rowlands, Casper da Costa-Luis, Alexander Schepanovski, Ivan Shcheklein, Batuhan Taskaya, Gao, Jorge Orpinel, David de la Iglesia Castro, Fábio Santos, Aman Sharma, Dave Berenbaum, Zhanibek, Dani Hodovic, Nikita Kodenko, Andrew Grigorev, Earl, daniele, Nabanita Dash, George Vyshnya, maykulkarni, Max Hora, Vera, Sanidhya Mangal, and Wojciech Baranowski. 2022\.  _DVC: Data Version Control - Git for Data & Models_.  https://doi.org/10.5281/zenodo.7039863
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep learning.  _nature_ 521, 7553 (2015), 436.
  * Lee et al. (2017) Kisuk Lee, Jonathan Zung, Peter Li, Viren Jain, and H Sebastian Seung. 2017.  Superhuman accuracy on the SNEMI3D connectomics challenge.  _arXiv preprint arXiv:1706.00120_ (2017).
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Luan et al. (2022) Frank Sifei Luan, Stephanie Wang, Samyukta Yagati, Sean Kim, Kenneth Lien, SangBin Cho, Eric Liang, and Ion Stoica. 2022.  Exoshuffle: Large-Scale Shuffle at the Application Level.  _arXiv preprint arXiv:2203.05072_ (2022).
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).
  * Miles et al. (2020) Alistair Miles, John Kirkham, Martin Durant, James Bourbeau, Tarik Onalan, Joe Hamman, Zain Patel, shikharsg, Matthew Rocklin, raphael dussin, Vincent Schut, Elliott Sales de Andrade, Ryan Abernathey, Charles Noyes, sbalmer, pyup.io bot, Tommy Tran, Stephan Saalfeld, Justin Swaney, Josh Moore, Joe Jevnik, Jerome Kelleher, Jan Funke, George Sakkis, Chris Barnes, and Anderson Banihirwe. 2020\.  _zarr-developers/zarr-python: v2.4.0_.  https://doi.org/10.5281/zenodo.3773450
  * Oliphant (2006) Travis E Oliphant. 2006\.  _A guide to NumPy_. Vol. 1.  Trelgol Publishing USA.
  * Oord et al. (2016) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ (2016).
  * Papadopoulos et al. (2016) Stavros Papadopoulos, Kushal Datta, Samuel Madden, and Timothy Mattson. 2016.  The tiledb array data storage manager.  _Proceedings of the VLDB Endowment_ 10, 4 (2016), 349–360.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in pytorch.  (2017).
  * Pedro Pedreira (2022) Masha Basmanova Kevin Wilfong Laith Sakka Krishna Pai Wei He Biswapesh Chattopadhyay Pedro Pedreira, Orri Erling. 2022\.  Velox: Meta’s Unified Execution Engine.  _Proceedings of the VLDB Endowment_ (2022).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Rajpurkar et al. (2017) Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. 2017\.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  _arXiv preprint arXiv:1711.05225_ (2017).
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021\.  Zero-shot text-to-image generation. In _International Conference on Machine Learning_. PMLR, 8821–8831.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016.  You only look once: Unified, real-time object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 779–788.
  * Sabne (2020) Amit Sabne. 2020\.  Xla: Compiling machine learning for peak performance.  (2020).
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022\.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  _arXiv preprint arXiv:2205.11487_ (2022).
  * Samwel et al. (2022) Tom van Bussel Samwel, Herman van Hovell, Maryann Xue, Reynold Xin, and Matei Zaharia. 2022\.  Photon: A Fast Query Engine for Lakehouse Systems.  (2022).
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al. 2022\.  LAION-5B: An open large-scale dataset for training next generation image-text models.  (2022).
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021\.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ (2021).
  * Schwan et al. (2003) Philip Schwan et al. 2003\.  Lustre: Building a file system for 1000-node clusters. In _Proceedings of the 2003 Linux symposium_ , Vol. 2003. 380–386.
  * Sethi et al. (2019) Raghav Sethi, Martin Traverso, Dain Sundstrom, David Phillips, Wenlei Xie, Yutian Sun, Nezih Yegitbasi, Haozhun Jin, Eric Hwang, Nileema Shingte, et al. 2019\.  Presto: SQL on everything. In _2019 IEEE 35th International Conference on Data Engineering (ICDE)_. IEEE, 1802–1813.
  * Shvachko et al. (2010) Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, et al. 2010\.  The hadoop distributed file system.. In _MSST_ , Vol. 10. 1–10.
  * Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2018\.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.  _Science_ 362, 6419 (2018), 1140–1144.
  * Stumpf et al. (2018) K Stumpf, S Bedratiuk, and O Cirit. 2018.  Michelangelo PyML: introducing Uber’s platform for rapid python ML model development.  _Uber. See: https://eng. uber. com/michelangelo-pyml_ (2018).
  * Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019\.  Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In _7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)_. Leibniz-Institut für Deutsche Sprache.
  * Team (2022) Squirrel Developer Team. 2022\.  Squirrel: A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.  _GitHub. Note: https://github.com/merantix-momentum/squirrel-core_ (2022).  https://doi.org/10.5281/zenodo.6418280
  * Thusoo et al. (2010) Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Antony, Hao Liu, and Raghotham Murthy. 2010.  Hive-a petabyte scale data warehouse using hadoop. In _2010 IEEE 26th international conference on data engineering (ICDE 2010)_. IEEE, 996–1005.
  * Tunyasuvunakool et al. (2021) Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Žídek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, et al. 2021\.  Highly accurate protein structure prediction for the human proteome.  _Nature_ 596, 7873 (2021), 590–596.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vohra (2016) Deepak Vohra. 2016\.  Apache parquet.  In _Practical Hadoop Ecosystem_. Springer, 325–335.
  * Wang et al. (2021) Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. 2021\.  Milvus: A Purpose-Built Vector Data Management System. In _Proceedings of the 2021 International Conference on Management of Data_. 2614–2627.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ 32 (2019).
  * Zaharia et al. (2010) Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010\.  Spark: Cluster computing with working sets.  _HotCloud_ 10, 10-10 (2010), 95.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification. In _Advances in neural information processing systems_. 649–657.


#  Natural hyperbolicity in the layered hexagonal crystal structure

Ali Ebrahimian<EMAIL_ADDRESS>School of Nano Science, Institute for
Research in Fundamental Sciences (IPM), Tehran 19395-5531, Iran  Reza Asgari
<EMAIL_ADDRESS>School of Physics, Institute for Research in Fundamental
Sciences (IPM), Tehran 19395-5531, Iran  ARC Centre of Excellence in Future
Low-Energy Electronics Technologies, UNSW Node, Sydney 2052, Australia

###### Abstract

Discovering the physical requirements for meeting the indefinite permittivity
in natural material as well as proposing a new natural hyperbolic media offer
a possible route to significantly improve our knowledge and ability to confine
and controlling light in optoelectronic devices. We demonstrate the
hyperbolicity in a class of materials with hexagonal P6/mmm and P63/mmc
layered crystal structures and its physical origin is thoroughly investigated.
By utilizing density functional theory and solving the Bethe-Salpeter equation
(BSE), we find that the layered crystal structure and symmetry imposed
constraints in Li3N gives rise to an exceedingly strong anisotropy in optical
responses along in- and out-of-plane directions of the crystals making it a
natural hyperbolic in a broad spectral range from the visible spectrum to the
ultraviolet. More excitingly, the hyperbolicity relation to anisotropic
interband absorption in addition to the impressive dependency of the
conduction band to the lattice constant along the out-of-plane direction
provide the hyperbolicity tunability in these hexagonal structures under
strain, doping, and alloying. Our findings not only suggest a large family of
real hexagonal compounds as a unique class of materials for realization of the
highly tunable broad band hyperbolicity but also offers an approach to search
for new hyperbolic materials.

##  I Introduction

Thanks to relative permittivity manipulation through materials engineering,
metamaterials have been the subject of intense interest in various science and
engineering research communities aiming to conduct their interaction with
electromagnetic and optical signals Poddubny  _et al._ (2013). The relative
permittivity, $\epsilon_{\rm 1}(\omega)$+i$\epsilon_{\rm 2}(\omega)$ is the
main optical quantity that provides the description of the interaction between
matter and electromagnetic waves. In hyperbolic metamaterials, $\epsilon_{\rm
1}(\omega)$ exhibits different signs along different directions revealing the
extreme anisotropy of the electronic structure. Anisotropic optical materials
exhibit numerous novel properties and potential applications, like negative
refraction Hoffman  _et al._ (2007); Yao  _et al._ (2008), subwavelength
imaging Liu  _et al._ (2007); Rho  _et al._ (2010); Lu and Sridhar (2008) in
the far-field, optical nanoscale cavity Yao  _et al._ (2011); Yang  _et al._
(2012).

The electronic structure anisotropy is typically achieved in artificially
engineered metamaterials suffering from complicated nanofabrication
requirements, the high degree of interface electron scatterings, and the
feature size of the components. Circumventing these limitations, natural
hyperbolic materials provide the extended hyperbolic frequency window with low
losses, high light confinement, and a larger photonics density of states Sun
_et al._ (2014a); Guan  _et al._ (2017); Ma  _et al._ (2018a); Li  _et al._
(2018a); Zheng  _et al._ (2019a); Edalati-Boostan  _et al._ (2020). Heretofore
several materials, like graphite, MgB2, cuprate, ruthenate,and tetradymite
have been predicted as natural hyperbolic materials Sun  _et al._ (2014b,
2011); Alekseyev  _et al._ (2012); Esslinger  _et al._ (2014); Korzeb  _et
al._ (2015), while experimentally they were observed only in few of them,
e.g., in MoO3 surfaces Ma  _et al._ (2018b); Zheng  _et al._ (2019b),
structured hexagonal boron nitride Li  _et al._ (2018b) and T${}_{d}\,$-WTe2
thin films Wang  _et al._ (2020). Previous materials investigations reveal
that the layered crystal structures exhibit excellent intrinsic out-of-plane
optical anisotropy providing the required conditions for achieving natural
hyperbolic materials Basov  _et al._ (2016); Low  _et al._ (2017); Narimanov
and Kildishev (2015).

In the same way, a recent study shows that the layered nature of group-I
nitrides with hexagonal P6/mmm ($\alpha$-type) crystal structure gives rise to
an exceedingly strong anisotropy in the electronic structure Ebrahimian  _et
al._ (2020). The $\alpha$-Li3N-type structure possesses a layered structure
composed of alternating planes of the hexagonal Li2N and pure Li+-ions with
weak interactions along the stacking direction. It should be noted that the
similar behavior has been also reported in its counterparts; the high-
temperature superconductor MgB2 and AlB2 Loa  _et al._ (2002); Cheng  _et al._
(2020). In particular, the weak Coulombic interactions between adjacent layers
compared with interactions in a layer leads to a less dispersive valence band
maximum (VBM) along the $\Gamma$-A. It enhances the anisotropy of the
electronic structures causing the highly anisotropic optical responses along
the in- and out-of-plane directions of the crystals. Furthermore, the binary
alkali pnictide A3B (A = Li, Na; B = N, P, As, Sb, Bi) with hexagonal P63/mmc
($\beta$-type) structure meets the similar layered structure and nearly flat
bands along the $\Gamma$-A which are responsible for the anisotropy of the
electronic structure Nie  _et al._ (2018); Jin  _et al._ (2020). At this
stage, we show, owing to these anisotropic responses, materials with $\alpha$
and $\beta$-type crystal structures belong to the rare class of natural
hyperbolic materials via thoroughly investigation of the electronic and
optical properties of $\alpha$-Li3N. We emphasize that all the $\alpha$ and
$\beta$-type structures have qualitatively the same physics as $\alpha$-Li3N
does (see supplemental materials).

The Li3N, as a prominent member of this class, has been extensively studied as
a distinguished superionic conductor while its optical properties remain
silent. Using first principle calculations and Bethe-Salpeter Equation
approach, we show that Li3N is a natural hyperbolic in a broad spectral range
from the visible to the ultraviolet spectrums. Interestingly, it exhibits type
II hyperbolic response in lower and upper parts of the hyperbolic window while
the middle one belongs to type I hyperbolic response. We find that the
hyperbolicity in Li3N is related to anisotropic interband absorption between
the valence and conduction bands imposed by symmetry constraints and
transition selection rules.

As a standard feature of this new class of natural hyperbolic materials
(Narimanov and Kildishev, 2015; Cheng  _et al._ , 2020; Nie  _et al._ , 2018),
the valence and conduction bandgap of Li3N is highly impressionable and
susceptible to the lattice constant c. These unique conditions provide the
possibility of the hyperbolicity tunability in Li3N with strain, doping and
alloying. In fact, our calculations show that the hyperbolic window of Li3N
and the relative intensity of the sub-diffraction and diffraction-limited
waves in transmission through Li3N can be efficiently tuned by applying strain
along the $z-$axis and doping in addition to using its ternary compounds like
Li2K(Na)N. Although Li3N and Li2K(Na)N possess the same structure, the larger
ionic radius of K(Na) leads to a band crossing along the $z-$direction
(Ebrahimian  _et al._ , 2020), which extends the hyperbolic window in the
visible regime highly demanding in light device applications. Beside Li3N and
the related alloys, the anisotropic electronic structure of $\alpha$ and
$\beta$-type compounds gives rise to hyperbolic frequency windows for both
type I and II as well offering hexagonal P6/mmm and P63/mmc layered structures
as a unique class of materials for realization of the highly tunable broadband
hyperbolicity and relevant device applications.

The structure of the paper is as follows. In Sec. II, we briefly introduce the
simulation methods. Sec. III is devoted to the numerical results of the study,
focusing on the electronic and optical properties of the $\alpha$-Li3N crystal
structure. Finally, we summarize our results in Sec. IV.

##  II Theoretical and Computational Methods

Figure 1: (Color online) The crystal structure (left panel) and the first
Brillouin zone (right panel) of $\alpha$-Li3N with P6/mmm symmetry. The mirror
reflection plane is shown by Mz.

The electronic and linear optical properties of the insulator crystal
structures (e.g. Li3N) are calculated in the framework of density functional
theory (DFT) and many-body perturbation theory. DFT is based on the accurate
all-electron full-potential linearized augmented plane wave (FP-LAPW) method
as implemented in Exciting code Gulans  _et al._ (2014). Ground-state
properties are calculated using the hybrid density functional (HSE06) Heyd
_et al._ (2003); Hummer  _et al._ (2009) in addition to PBE-type Perdew  _et
al._ (1996) generalized gradient approximation functional to meet adequately
high accuracy requirements. The hole doping induced in the $\alpha$-Li3N is
simulated by shifting downward the Fermi level according to a standard rigid-
band model. A 20$\times$20$\times$20 Monkhorst-pack k-point mesh is properly
used in the computations. For Li3N, the optical spectra are obtained by
solving the Bethe-Salpeter equation (BSE); the equation of motion of the two-
particle Green’s function Hanke and Sham (1988); Strinati (1988). The matrix
eigenvalue form of the BSE is thus given by Onida  _et al._ (2002); Puschnig
and Ambrosch-Draxl (2002); Rohlfing and Louie (2000); Ebrahimian and Dadsetani
(2017)

$\sum_{\rm\nu^{\prime}c^{\prime}k^{\prime}}{\cal H}^{{\text{BSE}}}_{\bf\nu
ck,\nu^{\prime}c^{\prime}k^{\prime}}{A}^{\lambda}_{\nu^{\prime}c^{\prime}{\bf
k}^{\prime}}={E}^{\lambda}{A}^{\lambda}_{\bf\nu ck}$ (1)

where $\nu$, $c$ and $k$ indicate the valence band, conduction band and
k-points in the reciprocal space, respectively. Eigenvalues ${E}^{\lambda}$
and eigenvectors ${A}^{\lambda}_{\bf\nu ck}$ represent the excitation energy
of the $j$th-correlated electron-hole pair and the coupling coefficients used
to construct the exciton wave function, respectively. In the Tamm-Dancoff
approximation the effective two-particle Hamiltonian for a spin-degenerate
system is given by

${\cal H}^{\text{BSE}}={\cal H}^{\text{diag}}+2{\cal H}^{x}+{\cal H}^{d}$ (2)

where the kinetic term ${\cal H}^{\text{diag}}$ is determined from vertical
transitions between noninteracting quasiparticle energy and corresponds to the
independent particle approximation. The screened Coulomb interaction, ${\cal
H}^{d}$ and the bare electron-hole exchange, ${\cal H}^{x}$ are responsible
for the formation of bound excitons. Having calculated the eigenvalues and
eigenvectors of the BSE, the long-wavelength limit of the imaginary part of
the dielectric function $\epsilon_{\rm ii}(\omega)$ is given by Onida  _et
al._ (2002); Ebrahimian and Dadsetani (2017)

$Im\epsilon_{\rm
ii}(\omega)=\frac{8{\pi}^{2}}{\Omega}\sum_{\rm\lambda}{{|t_{\rm\lambda}|}^{2}\delta(\omega-E^{\lambda})}$
(3) $t_{\rm\lambda}=\sum_{\rm\nu c{\bf k}}{{A}^{\lambda}_{\nu c{\bf
k}}\frac{<\nu k|\hat{p}|ck>}{\varepsilon^{\rm c}_{\bf
k}-\varepsilon^{\nu}_{\bf k}}}$ (4)

where $\varepsilon^{\rm l}_{\bf k}$ with $l=c,\nu$ is the eigenvalue of the
Kohn-Sham (KS) equation. The imaginary part of the DFT dielectric function
shows the excitations in the system. The optical properties of semimetal
structures like doped Li3N, Li2KN and Na3N are considered using Time-dependent
density-functional theory (TDDFT) Gross and Kohn (1985). The inverse
dielectric matrix $\epsilon$ is related to the susceptibility $\chi$ (Ref.
Onida  _et al._ (2002); Yan  _et al._ (2011)) by the relation
$\epsilon^{-1}(q,\omega)=1+v(q)\chi(q,\omega)$, where $v(q)$ is the bare
Coulomb potential. In TDDFT the susceptibility $\chi(q,\omega)$ is obtained by
making use of the linear response to TDDFT through the solution of the Dyson
equation;

$\chi(q,\omega)={\chi}_{\rm 0}(q,\omega)+{\chi}_{\rm
0}(q,\omega)(v(q)+f_{xc}(q))\chi(q,\omega)$ (5)

where $\chi_{\rm 0}(q,\omega)$ is the KS susceptibility expressed in terms of
the KS eigenvalues and eigenfunctions, while $f_{xc}(q)$ is the exchange-
correlation kernel, for which we make use of the adiabatic local-density
approximation (ALDA). The neglect of the $f_{xc}(q)$ term corresponds to the
random-phase approximation (RPA). The macroscopic dielectric function
$\epsilon_{M}$ is therefore defined as

$\epsilon_{M}({\bf q}^{\prime},\omega)=\lim_{q\to
0}{\frac{1}{\epsilon^{-1}_{\bf G=0,G^{\prime}=0}({\bf q},\omega)}}$ (6)

where ${\bf q}^{\prime}={\bf q}+{\bf G}$ and ${\bf q}$ represents the momenta
inside the first Brillouin zone.

Figure 2: (Color online) The calculated bulk electronic band structure of (a)
Li3N and (b) Na3N (in PBE). The value of the Fermi energy for 0.1 holes/u.c is
shown by a green solid line. The PBE gap of Li3N is 1.04 eV while the gap is
2.12 eV within the HSE06. Except the bandgap value, the structures calculated
by HSE06 are remarkably similar to those obtained within the PBE in the
vicinity of the Fermi energy.

##  III  Numerical Results and Discussions

Lithium nitrite ($\alpha$-Li3N) crystallizes in the hexagonal P6/mmm structure
($\alpha$-Li3N-type crystal structure) as shown in Fig. 1. The $\alpha$-Li3N
structure possesses a layered structure composed of alternating planes of the
hexagonal Li2N and pure Li+-ions. In the Li2N layers each N (0, 0, 0) is at
the center of a symmetrical hexagon formed by the six neighboring Li-ions
(1/3, 2/3, 0) and (2/3, 1/3, 0) in units of lattice vectors. From now on,
these Li atoms denote as Li(1) and the Li atom above the N atoms consider as
Li(2). In this structure, the three lithium atoms donating their 2s electrons
to the nitrogen, resulting in Li+-ions and an N3--ion. For the Li3N crystal
structure, we obtain the lattice constant a= b= 3.648 and c= 3.885 Å which are
completely consistent with the ones reported in experiment Nazri  _et al._
(1994); Beister  _et al._ (1988); Vajenine  _et al._ (2009).

Figure 3: (Color online) The imaginary (a, b) and real (c, d) parts of the in-
plane (top) and the out-of-plane (bottom) components of the macroscopic
dielectric function Li3N. The optical spectra of both the
Im$\epsilon_{ii}(\omega)$ and Re$\epsilon_{ii}(\omega)$ show considerable
anisotropy between the in-plane and out-of-plane components with distinct
resonances in various directions. The polarization dependence originates from
the structural anisotropy of Li3N. The BSE results are qualitatively different
from those obtained within the RPA especially in the lower part of the
absorption. Moreover, the static dielectric constants along the in-plane and
out-of-plane are 5.78 and 5.64 for Li3N, respectively.

The calculated orbital resolved band structure of Li3N shows that the
conduction band minimum (CBM) around the $\Gamma$ point is parabolic and
mainly contributed by Li-s orbital while the valence band maximum (VBM) along
the $\Gamma$-A is a doubly-degenerate band and mostly composed of N-p orbital.
The weak interactions along the $z$ direction lead to a less dispersive pz VBM
along the $\Gamma$-A and enhances the anisotropy of the electronic structures.
This is the common feature of the $\alpha$ and $\beta$-type crystals and
response structure for the anisotropic optical properties.

As this novel natural hyperbolic class of materials contains also the
semimetals with a Dirac node along the $\Gamma$-A, we depicte the band
structure of Na3N as a semimetal member example in Fig. 2; providing a general
overview of these electronic structures. In spite of the crystal structure
similarity between Na3N and Li3N, as shown in Fig. 2, the CBM crosses the
doubly-degenerate VBM along the $\Gamma$-A generating a triply degenerate
nodal point (TDNP). However, due to similarities of optical properties of Li3N
and Na3N, we focus on the Li3N structure. The results for Na3N and some other
members are given in supplemental materials. In Li3N, the PBE gap is 1.04 eV
while the HSE06 gap is 2.12 eV otherwise the band structures calculated by
HSE06 are remarkably similar to those obtained within the PBE in the vicinity
of the Fermi energy (see Fig. 2(a)). The dielectric tensor of the hexagonal
Li3N is diagonal and retains two independent components; $\epsilon_{xx}$ =
$\epsilon_{yy}$ parallel to Li2N plane (in-plane) and $\epsilon_{zz}$
perpendicular to the Li2N plane (out-of-plane).

The BSE (RPA) computed optical spectra of Li3N is illustrated in Fig. 3. These
calculations are performed by including HSE06 correction to PBE one. In
general comparison, the optical spectra of both the Im$\epsilon_{ii}$ and
Re$\epsilon_{ii}$ show considerable anisotropy between the in-plane and out-
of-plane components with distinct resonances in various directions. The
polarization dependence of the optical absorption originates from the
structural anisotropy of Li3N. In fact, the symmetry and space orientation of
the valence and conduction bands would impose constraints on the optical
transition selection rules Hotta (2018) causing the polarization dependence of
the optical absorption. The strength of the interband transition contribution
to Im$\epsilon_{ii}$ is proportional to the product of the joint density of
states (JDOS) and the electric dipole transition moment matrix
$<\psi_{\nu}|\hat{r}|\psi_{\rm c}>$, where $\psi_{\nu}$ and $\psi_{\rm c}$ are
the wavefunctions of the valence and conduction bands, respectively and $r$ is
the position operator.

Figure 4: (Color online) (a) The real part of the dielectric tensor and (b)
the energy loss function to in-plane ($x-$polarized) and out-of-plane
($z-$polarized) incident light for Li3N. The spectral region for type I and II
hyperbolicity are marked with green and yellow background, respectively. Three
type-II hyperbolic frequency regimes in the visible region (from 3.1 eV to
3.54 eV) and ultraviolet (UV) regime including a narrow (from 8.23 eV to 8.5
eV) and a wide (from 8.67 eV to 12.48 eV) window. Interband transition peaks
in panel (b) are related to the peaks of Im$\epsilon_{ii}(\omega)$ while the
plasmon peaks correspond to zeros of the Re$\epsilon_{ii}$. Notice that the
plasmon excitation is located out of hyperbolic windows.

As the VBM is mainly composed of N-p orbital and the CBM is constructed from
Li-s orbital, the electric dipole transition moment matrix is $<s|\hat{r}|p>$.
The nonzero elements, and the electric dipole transition are determined by the
optical selection rules. At the $\Gamma$ point, the parity selection rule is
fulfilled owing to the opposite parities of the VBM and CBM. As the crystal
structure has a mirror symmetry, Rz= $\sigma_{\rm z}$, the VBM and CBM have
the same eigenvalues with respect to the mirror reflection operation Rz at the
$\Gamma$ point.

Figure 5: (Color online) The imaginary (a, b) and real parts (c, d) of the
dielectric tensor to the in-plane ($x$-polarized) and out-of-plane
($z$-polarized) incident light for +6$\%$ (a, c) and -6$\%$(b, d) strained
Li3N under tensile-strain along the $z$ axis. The spectral region for type I
and II hyperbolicity are marked with the green and yellow background,
respectively. The bandgap becomes smaller in +6$\%$ strained Li3N (a, c) so
the onset of the hyperbolic region moves to lower energy and gets inside the
visible region. In addition, the hyperbolic windows are red-shifted and become
broader (narrower) in the visible (UV) region by increasing (decreasing) the
lattice constant $c$.

Therefore, at the $\Gamma$ point the electric dipole transition is symmetry
allowed for the in-plane polarized light, while the transition in the out-of-
plane direction is forbidden under the mirror symmetry Rz. These symmetry
constraints are responsible for an anisotropy between the in-plane and out-of-
plane components of the dielectric tensors. The Im$\epsilon_{xx}$ has two main
structures above 2 and 6 eV while the Im$\epsilon_{zz}$ has one structure
above 2 eV and decays more rapidly. As a result, the Re$\epsilon_{xx}$
oscillates between positive and negative values providing the required
conditions for achieving a hyperbolic regime. Interestingly, Li3N is
transparent in most parts of the UV spectrum for the out-of-plane polarization
direction.

Figure 6: (Color online) (a) The calculated bulk electronic band structure and
(c) the energy loss function to the in-plane ($x$-polarized) and out-of-plane
($z$-polarized) incident light for Li2KN. (b) The imaginary and (d) real parts
of the dielectric tensor to the in-plane ($x$-polarized) and out-of-plane
($z$-polarized) incident light for Li2KN. The spectral region for type I and
II hyperbolicity are marked with the green and yellow background,
respectively. Type-I hyperbolic frequency regime is broader than type-II
hyperbolic window with a hyperbolicity onset in the visible region at 2.3 eV.
The transition from type I to type II occurs at 2.79 eV where Re$\epsilon_{\rm
xx}(\omega)$ and Re$\epsilon_{\rm zz}(\omega)$ vanish with an opposite sign
slope. Furthermore, the loss function components show small values meaning
that there is no energy loss when the material is hyperbolic.

As shown in Fig. 3, the BSE results are qualitatively different from those
obtained within the RPA especially in the lower part of the absorption where
the absorption onset is dominated by two intense peaks formed by bound
excitons with large oscillator strengths. The analysis of the k-space
distribution of the exciton weights shows that the first excitation originates
from the VBM to the conduction band at the $\Gamma$ point where the lowest-
energy excitation is produced by the transition between the highest occupied
band and the lowest-unoccupied band at the $\Gamma$ point. The next
excitations are mainly constituted by transitions between the first (second)
highest occupied band and the lowest unoccupied band along the path $\Gamma$-A
($\Gamma$-K), close to the $\Gamma$ point. Moreover, the static dielectric
constants along the in-plane and out-of-plane are 5.78 and 5.64 for Li3N,
respectively. In considering the Re$\epsilon_{ii}$ of Li3N, Fig. 3(c, d),
shows the negative values of the out-of-plane component of the real part of
the dielectric function only from 3.94 up to 7.66 eV which correspond to a
high reflectivity region. Most excitingly, the in-plane component of
Re$\epsilon_{ii}$ is mostly positive in this region while it goes through a
high reflectivity region in outside of the region. As the in-plane and out-of-
plane components of Re$\epsilon_{ii}$ cross zero at different frequencies, the
Li3N fulfills the necessary conditions for type I (Re$\epsilon_{xx}$ $>$ 0 and
Re$\epsilon_{zz}$ $<$ 0) and II (Re$\epsilon_{xx}$ $<$ 0 and Re$\epsilon_{zz}$
$>$ 0) indefinite permittivity media.

There are three clear type-II hyperbolic frequency regimes in the visible
region (from 3.1 eV to 3.54 eV) and ultraviolet (UV) regime including a narrow
(from 8.23 eV to 8.5 eV) and a wide (from 8.67 eV to 12.48 eV) window as shown
in Fig. 4(a). The wide type-I hyperbolic frequency regime is located between
4.51 eV and 7.66 eV. Like type-I hyperbolic window supports propagating
diffraction-limited and sub-diffraction waves while the type II hyperbolic
region only supports the latter one. The relative intensity of the sub-
diffraction and diffraction-limited waves in transmission through Li3N can be
tuned by modifying the type of the hyperbolic window providing a unique
platform to the definitive experimental study of the exotic phenomena like the
anti-cutoff Korobkin  _et al._ (2010). An ideal hyperbolic material should be
nearly lossless in the hyperbolic window.

We show, on the other hand, the energy loss function of Li3N in Fig. 4(b). The
energy loss function, L($\epsilon(q,\omega)$) = Im (-1/$\epsilon(q,\omega)$),
which determined the energy loss of a fast electron moving across a medium, is
a complicated mixture of interband transitions and plasmons. Interband
transition peaks are related to the peaks of Im$\epsilon_{ii}$ while the
plasmon peaks correspond to zeros of the Re$\epsilon_{ii}$. The plasmon peak
of the loss function is larger if Re$\epsilon_{ii}$ is zero and
Im$\epsilon_{ii}$ is a significantly small value. As shown in Fig. 4(b), both
the components of the loss function show negligible values in the entire
hyperbolic energy window except a broad plasmon peak with a small intensity
above 7.5 eV. This plasmon excitation is actually located out of hyperbolic
windows. Therefore, Li3N is a low-loss in hyperbolic frequency regimes making
it an ideal natural hyperbolic material. The origins of the hyperbolic
response regimes can be detected by considering the optical transitions
between the valence and conduction bands in the hyperbolicity windows, shown
in the electronic band structure in Fig. 3.

Figure 7: (Color online) (a) The imaginary and (b) real parts of the
dielectric tensor to the in-plane ($x$-polarized) and out-of-plane
($z$-polarized) incident light for hole-doped (0.1 holes/u.c.) Li3N. The
spectral region for type I and II hyperbolicity are marked with the green and
yellow background, respectively. (c) The energy loss function for in-plane and
out-of-plane incident light. Under hole doping the $x-$polarized interband
absorption above 2 and 6 eV becomes weaker and broader than corresponding ones
in pristine Li3N. The negative values of the in-plane component of the real
part of the dielectric function is less pronounced in hole-doped. The loss
function reflects the anisotropic electronic structures. Up to 5 eV the out-
of-plane component of the loss function is nearly zero while the in-plane
component contains two lesser peaks.

Owing to the Kramers-Kronig relation Singh (2006) between the real and
imaginary parts of the dielectric function, the high oscillator strength
resonances of Im$\epsilon_{xx}$ and Im$\epsilon_{zz}$ at 2.8 eV and 3.7 eV are
responsible for the negative permittivity in Re$\epsilon_{xx}$ and
Re$\epsilon_{zz}$ at 3.1 eV and 3.9 eV in Fig. 4(a). These intense peaks in
Im$\epsilon_{xx}$ and Im$\epsilon_{zz}$ are related to the interband
transitions between the valence and conduction bands along the $\Gamma$-A and
the L-A (see Fig. 2). The band structure of the system determines the occupied
and unoccupied bands nearly parallel to each other in these domains and
therefore have the same dispersions, resulting in a constant energy difference
between the conduction and valence bands (${|\bigtriangledown_{{\bf k}}(E_{\rm
c}-E_{\nu})|}=0$) Wang and Low (2020). As the JDOS can be expressed in terms
of ${|\bigtriangledown_{{\bf k}}(E_{\rm c}-E_{\nu})|}$ Wang and Low (2020):

$\frac{1}{{2\pi}^{2}}\int\frac{d{S_{k}}}{|\bigtriangledown_{{\bf k}}(E_{\rm
c}-E_{\nu})|}$ (7)

where ${S_{k}}$ is the constant energy surface defined by ${E_{\rm
c}-E_{\nu}}={\text{const.}}$), thereby, the JDOS and absorption are
significantly large along the $\Gamma$-A and L-A directions.

The same as before, the polarization dependence of the negative permittivity
window (absorption) along the $\Gamma$-A and L-A directions is dictated by
symmetry constraints and transition selection rules. The point group along the
high symmetry lines, the $\Gamma$-A and L-A, are D6h and C2v which have
horizontal mirror plane $\sigma_{\rm d}$ (Rz = $\sigma_{\rm z}$) and vertical
mirror plane $\sigma_{\nu}$, respectively. The VBM and CBM have the same
eigenvalues with respect to the mirror reflection operation Rz along the
$\Gamma$-A direction while they maintain the same parity with respect to the
vertical mirror plane $\sigma_{\rm v}$ along the L-A. These constraints make
the $z-$polarized transitions ($x-$polarized transitions) forbidden along the
$\Gamma$-A (L-A) causing the strong anisotropic dielectric tensors which
represent the required condition for the hyperbolicity.

Furthermore, under tensile-strain along the $z-$axis, the bandgap of Li3N
becomes smaller so the onset of the hyperbolic region moves to lower energy
and gets inside the visible region as described in Fig. 5. In addition, the
hyperbolic windows are red-shifted and become broader (narrower) in the
visible (UV) region by increasing (decreasing) the lattice constant $c$.
Inversely, under the compressive strain, the spectral range of hyperbolicity
moves to the UV region and becomes smaller due to the bandgap increasing.

Having extensively explored the optical properties of the Li3N, we now
investigate the possible existence of the hyperbolicity in related alloys like
Li2KN crystal structures. The lattice constants and internal coordinates of
Li2KN were fully optimized and the obtained results (a= b= 3.713 and c= 5.701
Å) are completely consistent with the previous report Schön  _et al._ (2000).
Our calculations show that replacing Li-ion on top of the N with Potassium (K)
atoms reduces the interactions between the N-pz and K-s orbitals leading to
the reversal of the band ordering between the conduction and valence bands at
the center of the first BZ of Li3N (see Fig. 6). In comparison to Li3N, the
spectral range of the hyperbolicity are smaller and located below 6 eV in the
Li2KN. In Li2KN, type-I hyperbolic frequency regime is broader than type-II
hyperbolic window with a hyperbolicity onset in the visible region at 2.3 eV
which is highly demanding in light device applications. The transition from
type I to type II indefinite permittivity media occurs at 2.79 eV where
Re$\epsilon_{\rm xx}$ and Re$\epsilon_{\rm zz}$ become zero with an opposite
sign slope. The same scenario happens at 4.2 eV and type II transforms into
type I indefinite permittivity media. The type I hyperbolic frequency regime
extends up to 5.3 eV. This interesting arrangement of various types of
hyperbolic regimes makes Li2KN an ideal tunable natural indefinite
permittivity media where the type of hyperbolicity can be selected by tuning
the energy of light. Furthermore, the loss function components show small
values which means that there is almost no energy loss when the material is
hyperbolic.

Finally, we consider the tunability of the optical properties of Li3N by
regulating its Fermi level with doping. We consider 0.1 holes per unit cell by
setting the Fermi energy to 0.17 eV below the VBM (see Fig. 2). As the Fermi
level crosses two valence bands along the perpendicular to the $z-$direction,
an intense peak appears in Im$\epsilon_{xx}$ at $\sim$ 0.1 eV (Fig. 7). In
comparison to pristine Li3N, under hole doping the $x-$polarized interband
absorption above 2 and 6 eV becomes weaker and broader than corresponding ones
in pristine Li3N. As a result, the negative values of the in-plane component
of the real part of the dielectric function is less pronounced in hole-doped
Li3N causing the blue shift in type-II hyperbolic regimes. Our results show
that the type-II hyperbolic is just appeared in the UV region (from 8.9 to
12.1). However, the main intense peak of Im$\epsilon_{zz}$ is preserved and
red-shifted in hole-doped Li3N. Therefore, the type-I hyperbolic frequency
regime is located between 3.8 eV and 7.2 eV. The loss function of a hole-doped
Li3N reflects perfectly the anisotropic electronic structures. Up to 5 eV the
out-of-plane component of the loss function is nearly zero while the in-plane
component has two small peaks. From 6 to 10 eV, the situation gets reversed
and the out-of-plane component of the loss function exhibits a small peak in
the vanishing region of the in-plane component. However, neglecting the peak
around 7 eV, the loss function is negligible up to 11 eV.

##  IV DISCUSSION AND CONCLUSION

Natural two-dimensional materials with anisotropic optical response are
considered promising candidates for hosting hyperbolic isofrequency surface.
In addition, two-dimensional materials are characterized with highly confined
and low-loss polaritons, that can be additionally tuned with field effect
techniques as well as with strain. In the present work, we provide a detailed
ab initio study on an anisotropic optical response in the hexagonal crystal
structures. Using density functional theory and symmetry analysis, we have
proposed $\alpha$\- and $\beta$-type hexagonal layered crystal structure as a
new class of hyperbolic materials where the hyperbolicity is arisen from the
weak Coulombic interactions between adjacent layers compared with interactions
within a specific layer leads in addition to symmetry constraints and
transition selection rules. A comprehensive study of optical properties
introduces this class of materials as a new natural hyperbolic material in
which the type of hyperbolicity can be selected by changing the frequency of
light. More excitingly, the hyperbolicity in these materials can be tuned
using lattice strain along the $z$-axis and doping as well. Furthermore, the
many-body effects on the dielectric functions are discussed. We have also
shown this class of materials are a low-loss in hyperbolic frequency regimes
making them ideal natural hyperbolic materials. Our findings here can be
explored by current experiments.

Ultimately, we would like to emphasize potential viable applications of
$\alpha$-Li3N-( $\beta$-Li3N-) type hexagonal layered crystal structures in
comparison with layered transition metal dichalcogenides (TMDs) Gjerding  _et
al._ (2017). First of all, as discussed in this paper, this new class of
hyperbolic materials exhibits both type I and type-II hyperbolicity over the
wide range of frequencies which rarely found in nature Gjerding  _et al._
(2017). As a result, the type of hyperbolicity can be selected by tuning the
energy of light. Secondly, a previous study shows that the low-losses as main
responsible of a large Purcell factor in the hyperbolicity regime only meet at
the metallic TMDs where the conduction bands being sufficiently separated from
other bands by finite energy bandgaps. However, our calculations show that
$\alpha$-Li3N-( $\beta$-Li3N-) type crystal structures, both metallic and
semiconductor ones, are nearly lossless in the hyperbolic windows. Most
importantly, in $\alpha$-Li3N-( $\beta$-Li3N-) type crystal structures the
impressive dependency of the conduction band to the lattice constant $c$
provide the hyperbolicity tunability in these hexagonal structures under
strain, doping, and alloying which are more plausible than heterostructures of
standard TMDs.

##  V Acknowledgments

A. E is supported by Iran Science Elites Federation and R. A is supported by
the Australian Research Council Centre of Excellence in Future Low-Energy
Electronics Technologies (project number CE170100039).

## References

  * Poddubny  _et al._ (2013) A. Poddubny, I. Iorsh, P. Belov, and Y. Kivshar, “Hyperbolic metamaterials,” Nature photonics 7, 948 (2013).
  * Hoffman  _et al._ (2007) A. J. Hoffman, L. Alekseyev, S. S. Howard, K. J. Franz, D. Wasserman, V. A. Podolskiy, E. E. Narimanov, D. L. Sivco, and C. Gmachl, “Negative refraction in semiconductor metamaterials,” Nature materials 6, 946 (2007).
  * Yao  _et al._ (2008) J. Yao, Z. Liu, Y. Liu, Y. Wang, C. Sun, G. Bartal, A. M. Stacy, and X. Zhang, “Optical negative refraction in bulk metamaterials of nanowires,” Science 321, 930 (2008).
  * Liu  _et al._ (2007) Z. Liu, H. Lee, Y. Xiong, C. Sun, and X. Zhang, “Far-field optical hyperlens magnifying sub-diffraction-limited objects,” science 315, 1686 (2007).
  * Rho  _et al._ (2010) J. Rho, Z. Ye, Y. Xiong, X. Yin, Z. Liu, H. Choi, G. Bartal, and X. Zhang, “Spherical hyperlens for two-dimensional sub-diffractional imaging at visible frequencies,” Nature communications 1, 1 (2010).
  * Lu and Sridhar (2008) W. T. Lu and S. Sridhar, “Superlens imaging theory for anisotropic nanostructured metamaterials with broadband all-angle negative refraction,” Physical Review B 77, 233101 (2008).
  * Yao  _et al._ (2011) J. Yao, X. Yang, X. Yin, G. Bartal, and X. Zhang, “Three-dimensional nanometer-scale optical cavities of indefinite medium,” Proceedings of the National Academy of Sciences 108, 11327 (2011).
  * Yang  _et al._ (2012) X. Yang, J. Yao, J. Rho, X. Yin, and X. Zhang, “Experimental realization of three-dimensional indefinite cavities at the nanoscale with anomalous scaling laws,” Nature Photonics 6, 450 (2012).
  * Sun  _et al._ (2014a) J. Sun, N. M. Litchinitser, and J. Zhou, “Indefinite by nature: from ultraviolet to terahertz,” ACS Photonics 1, 293 (2014a).
  * Guan  _et al._ (2017) S. Guan, S. Y. Huang, Y. Yao, and S. A. Yang, “Tunable hyperbolic dispersion and negative refraction in natural electride materials,” Physical Review B 95, 165436 (2017).
  * Ma  _et al._ (2018a) W. Ma, P. Alonso-González, S. Li, A. Y. Nikitin, J. Yuan, J. Martín-Sánchez, J. Taboada-Gutiérrez, I. Amenabar, P. Li, S. Vélez, _et al._ , “In-plane anisotropic and ultra-low-loss polaritons in a natural van der waals crystal,” Nature 562, 557 (2018a).
  * Li  _et al._ (2018a) P. Li, I. Dolado, F. J. Alfaro-Mozaz, F. Casanova, L. E. Hueso, S. Liu, J. H. Edgar, A. Y. Nikitin, S. Vélez, and R. Hillenbrand, “Infrared hyperbolic metasurface based on nanostructured van der waals materials,” Science 359, 892 (2018a).
  * Zheng  _et al._ (2019a) Z. Zheng, N. Xu, S. L. Oscurato, M. Tamagnone, F. Sun, Y. Jiang, Y. Ke, J. Chen, W. Huang, W. L. Wilson, _et al._ , “A mid-infrared biaxial hyperbolic van der waals crystal,” Science advances 5, eaav8690 (2019a).
  * Edalati-Boostan  _et al._ (2020) S. Edalati-Boostan, C. Cocchi, and C. Draxl, “$\mathrm{Mo}$${\mathrm{te}}_{2}$ as a natural hyperbolic material across the visible and the ultraviolet region,” Phys. Rev. Materials 4, 085202 (2020).
  * Sun  _et al._ (2014b) J. Sun, N. M. Litchinitser, and J. Zhou, “Indefinite by nature: from ultraviolet to terahertz,” ACS Photonics 1, 293 (2014b).
  * Sun  _et al._ (2011) J. Sun, J. Zhou, B. Li, and F. Kang, “Indefinite permittivity and negative refraction in natural material: graphite,” Applied Physics Letters 98, 101901 (2011).
  * Alekseyev  _et al._ (2012) L. V. Alekseyev, V. A. Podolskiy, and E. E. Narimanov, “Homogeneous hyperbolic systems for terahertz and far-infrared frequencies.” Advances in OptoElectronics (2012), 0.1155/2012/267564.
  * Esslinger  _et al._ (2014) M. Esslinger, R. Vogelgesang, N. Talebi, W. Khunsin, P. Gehring, S. De Zuani, B. Gompf, and K. Kern, “Tetradymites as natural hyperbolic materials for the near-infrared to visible,” Acs Photonics 1, 1285 (2014).
  * Korzeb  _et al._ (2015) K. Korzeb, M. Gajc, and D. A. Pawlak, “Compendium of natural hyperbolic materials,” Optics express 23, 25406 (2015).
  * Ma  _et al._ (2018b) W. Ma, P. Alonso-González, S. Li, A. Y. Nikitin, J. Yuan, J. Martín-Sánchez, J. Taboada-Gutiérrez, I. Amenabar, P. Li, S. Vélez, C. Tollan, Z. Dai, Y. Zhang, S. Sriram, K. Kalantar-Zadeh, S.-T. Lee, R. Hillenbrand, and Q. Bao, “In-plane anisotropic and ultra-low-loss polaritons in a natural van der waals crystal,” Nature 562, 557 (2018b).
  * Zheng  _et al._ (2019b) Z. Zheng, N. Xu, S. L. Oscurato, M. Tamagnone, F. Sun, Y. Jiang, Y. Ke, J. Chen, W. Huang, W. L. Wilson, A. Ambrosio, S. Deng, and H. Chen, “A mid-infrared biaxial hyperbolic van der waals crystal,” Science Advances 5, eaav8690 (2019b).
  * Li  _et al._ (2018b) P. Li, I. Dolado, F. J. Alfaro-Mozaz, F. Casanova, L. E. Hueso, S. Liu, J. H. Edgar, A. Y. Nikitin, S. Vélez, and R. Hillenbrand, “Infrared hyperbolic metasurface based on nanostructured van der waals materials,” Science 359, 892 (2018b).
  * Wang  _et al._ (2020) C. Wang, S. Huang, Q. Xing, Y. Xie, C. Song, F. Wang, and H. Yan, “Van der waals thin films of wte 2 for natural hyperbolic plasmonic surfaces,” Nature communications 11, 1 (2020).
  * Basov  _et al._ (2016) D. Basov, M. Fogler, and F. G. De Abajo, “Polaritons in van der waals materials,” Science 354 (2016), 10.1126/science.aag1992.
  * Low  _et al._ (2017) T. Low, A. Chaves, J. D. Caldwell, A. Kumar, N. X. Fang, P. Avouris, T. F. Heinz, F. Guinea, L. Martin-Moreno, and F. Koppens, “Polaritons in layered two-dimensional materials,” Nature materials 16, 182 (2017).
  * Narimanov and Kildishev (2015) E. E. Narimanov and A. V. Kildishev, “Naturally hyperbolic,” Nature Photonics 9, 214 (2015).
  * Ebrahimian  _et al._ (2020) A. Ebrahimian, R. Asgari, and M. Dadsetani, “Topological phases in $\alpha$-li3n-type crystal structure of light-element compounds,” arXiv:2009.11941 (2020).
  * Loa  _et al._ (2002) I. Loa, K. Kunc, K. Syassen, and P. Bouvier, “Crystal structure and lattice dynamics of alb 2 under pressure and implications for mgb 2,” Physical Review B 66, 134101 (2002).
  * Cheng  _et al._ (2020) C. Cheng, M.-Y. Duan, Z. Wang, and X.-L. Zhou, “Alb2 and mgb2: a comparative study of their electronic, phonon and superconductivity properties via first principles,” Philosophical Magazine , 1 (2020).
  * Nie  _et al._ (2018) T. Nie, L. Meng, Y. Li, Y. Luan, and J. Yu, “Phase transition studies of na3bi system under uniaxial strain,” Journal of Physics: Condensed Matter 30, 125502 (2018).
  * Jin  _et al._ (2020) L. Jin, X. Zhang, T. He, W. Meng, X. Dai, and G. Liu, “Electronic structure, doping effect and topological signature in realistic intermetallics li 3- x na x m (x= 3, 2, 1, 0; m= n, p, as, sb, bi),” Physical Chemistry Chemical Physics 22, 5847 (2020).
  * Gulans  _et al._ (2014) A. Gulans, S. Kontur, C. Meisenbichler, D. Nabok, P. Pavone, S. Rigamonti, S. Sagmeister, U. Werner, and C. Draxl, “Exciting: a full-potential all-electron package implementing density-functional theory and many-body perturbation theory,” Journal of Physics: Condensed Matter 26, 363202 (2014).
  * Heyd  _et al._ (2003) J. Heyd, G. E. Scuseria, and M. Ernzerhof, “Hybrid functionals based on a screened coulomb potential,” The Journal of chemical physics 118, 8207 (2003).
  * Hummer  _et al._ (2009) K. Hummer, J. Harl, and G. Kresse, “Heyd-scuseria-ernzerhof hybrid functional for calculating the lattice dynamics of semiconductors,” Physical Review B 80, 115205 (2009).
  * Perdew  _et al._ (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, “Generalized gradient approximation made simple,” Physical review letters 77, 3865 (1996).
  * Hanke and Sham (1988) W. Hanke and L. J. Sham, “Density-functional theory in insulators: Analytical model for $\sigma$ xc, v xc, and the gap correction,” Physical Review B 38, 13361 (1988).
  * Strinati (1988) G. Strinati, “Application of the green’s functions method to the study of the optical properties of semiconductors,” La Rivista del Nuovo Cimento (1978-1999) 11, 1 (1988).
  * Onida  _et al._ (2002) G. Onida, L. Reining, and A. Rubio, “Electronic excitations: density-functional versus many-body green’s-function approaches,” Reviews of modern physics 74, 601 (2002).
  * Puschnig and Ambrosch-Draxl (2002) P. Puschnig and C. Ambrosch-Draxl, “Optical absorption spectra of semiconductors and insulators including electron-hole correlations: An ab initio study within the lapw method,” Physical Review B 66, 165105 (2002).
  * Rohlfing and Louie (2000) M. Rohlfing and S. G. Louie, “Electron-hole excitations and optical spectra from first principles,” Physical Review B 62, 4927 (2000).
  * Ebrahimian and Dadsetani (2017) A. Ebrahimian and M. Dadsetani, “Dependence of topological and optical properties on surface-terminated groups in two-dimensional molybdenum dinitride and tungsten dinitride nanosheets,” Physical Chemistry Chemical Physics 19, 30301 (2017).
  * Gross and Kohn (1985) E. K. U. Gross and W. Kohn, “Local density-functional theory of frequency-dependent linear response,” Physical review letters 55, 2850 (1985).
  * Yan  _et al._ (2011) J. Yan, J. J. Mortensen, K. W. Jacobsen, and K. S. Thygesen, “Linear density response function in the projector augmented wave method: Applications to solids, surfaces, and interfaces,” Physical Review B 83, 245122 (2011).
  * Nazri  _et al._ (1994) G. Nazri, C. Julien, and H. Mavi, “Structure of li3x (x= n, p, as) superionic conductors: X-ray diffraction and ftir studies,” Solid state ionics 70, 137 (1994).
  * Beister  _et al._ (1988) H. J. Beister, S. Haag, R. Kniep, K. Strössner, and K. Syassen, “Phase transformations of lithium nitride under pressure,” Angewandte Chemie International Edition in English 27, 1101 (1988).
  * Vajenine  _et al._ (2009) G. V. Vajenine, X. Wang, I. Efthimiopoulos, S. Karmakar, K. Syassen, and M. Hanfland, “Structural phase transitions of sodium nitride at high pressure,” Physical Review B 79, 224107 (2009).
  * Hotta (2018) S. Hotta,  _Mathematical Physical Chemistry_ (Springer, 2018).
  * Korobkin  _et al._ (2010) D. Korobkin, B. Neuner, C. Fietz, N. Jegenyes, G. Ferro, and G. Shvets, “Measurements of the negative refractive index of sub-diffraction waves propagating in an indefinite permittivity medium,” Optics express 18, 22734 (2010).
  * Singh (2006) J. Singh,  _Optical properties of condensed matter and applications_ , Vol. 6 (John Wiley & Sons, 2006).
  * Wang and Low (2020) H. Wang and T. Low, “Hyperbolicity in two-dimensional transition metal ditellurides induced by electronic bands nesting,” Phys. Rev. B 102, 241104(R) (2020).
  * Schön  _et al._ (2000) J. Schön, M. Wevers, and M. Jansen, “Investigation of the possible ternary nitrides in the system li3n/na3n,” Solid state sciences 2, 449 (2000).
  * Gjerding  _et al._ (2017) M. N. Gjerding, R. Petersen, T. G. Pedersen, N. A. Mortensen, and K. S. Thygesen, “Layered van der waals crystals with hyperbolic light dispersion,” Nature communications 8, 1 (2017).