---
language: en
tags:
- tokenizer
- pytorch
- streaming
library_name: nano
pipeline_tag: token-classification
datasets:
- Salesforce/wikitext
---

# Nano Tokenizer

This tokenizer was trained using a Python-only pipeline (no `transformers` or `tokenizers`), on a dataset streamed from the Hugging Face Hub.

## Usage

```python
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained("goabonga/wikitext-2-raw-v1")
```