Hub documentation

Daft

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Daft

Daft is a high-performance data engine providing simple and reliable data processing for any modality and scale. Daft has native support for reading from and writing to Hugging Face datasets.

Getting Started

To get started, pip install daft with the huggingface feature:

pip install 'daft[huggingface]'

Read

Daft is able to read datasets directly from the Hugging Face Hub using the daft.read_huggingface() function or via the hf://datasets/ protocol.

Reading an Entire Dataset

Using daft.read_huggingface(), you can easily load a dataset.

import daft

df = daft.read_huggingface("username/dataset_name")

This will read the entire dataset into a DataFrame.

Reading Specific Files

Not only can you read entire datasets, but you can also read individual files from a dataset repository. Using a read function that takes in a path (such as daft.read_parquet(), daft.read_csv(), or daft.read_json()), specify a Hugging Face dataset path via the hf://datasets/ prefix:

import daft

# read a specific Parquet file
df = daft.read_parquet("hf://datasets/username/dataset_name/file_name.parquet")

# or a csv file
df = daft.read_csv("hf://datasets/username/dataset_name/file_name.csv")

# or a set of Parquet files using a glob pattern
df = daft.read_parquet("hf://datasets/username/dataset_name/**/*.parquet")

Write

Daft is able to write Parquet files to a Hugging Face dataset repository using daft.DataFrame.write_huggingface. Daft supports Content-Defined Chunking and Xet for faster, deduplicated writes.

Basic usage:

import daft

df: daft.DataFrame = ...

df.write_huggingface("username/dataset_name")

See the DataFrame.write_huggingface API page for more info.

Authentication

The token parameter in daft.io.HuggingFaceConfig can be used to specify a Hugging Face access token for requests that require authentication (e.g. reading private dataset repositories or writing to a dataset repository).

Example of loading a dataset with a specified token:

from daft.io import IOConfig, HuggingFaceConfig

io_config = IOConfig(hf=HuggingFaceConfig(token="your_token"))
df = daft.read_parquet("hf://datasets/username/dataset_name", io_config=io_config)
< > Update on GitHub