Upload tokenizer files (vocab, config, README)
Browse files- README.md +3 -0
- vocab.json +0 -0
README.md
CHANGED
|
@@ -5,6 +5,9 @@ tags:
|
|
| 5 |
- pytorch
|
| 6 |
- streaming
|
| 7 |
library_name: nano
|
|
|
|
|
|
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
# Nano Tokenizer
|
|
|
|
| 5 |
- pytorch
|
| 6 |
- streaming
|
| 7 |
library_name: nano
|
| 8 |
+
pipeline_tag: token-classification
|
| 9 |
+
datasets:
|
| 10 |
+
- wikitext/wikitext-2-raw-v1
|
| 11 |
---
|
| 12 |
|
| 13 |
# Nano Tokenizer
|
vocab.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|