metadata
license: apache-2.0
The Hugging Face fast tokenizer for LLM-jp ABCI challenge 2023.
The vocab size is 96,868.
Requirements:
- transformers>=4.34.0
- tokenizers>=0.14.0
- torch
Usage:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v22b2")