---
license: apache-2.0
---
The Hugging Face fast tokenizer for LLM-jp ABCI challenge 2023.

The vocab size is 96,867.

Requirements:
- transformers>=4.34.0
- tokenizers>=0.14.0
- torch

Usage:
```Python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v22b2")
```