tangledgroup
/

tangled-alpha-0.12-core

Text Generation

Model card Files Files and versions

mtasic85 commited on Apr 5

Commit

ad3a56c

·

1 Parent(s): f1c3ec3

tokenizer

Files changed (1) hide show

scripts/train_tokenizer.py +2 -0

scripts/train_tokenizer.py CHANGED Viewed

@@ -44,6 +44,8 @@ special_tokens = [
     '</tool_call>',
     '<tool_response>',
     '</tool_response>',
     '<think>',
     '</think>',
     '<answer>',

     '</tool_call>',
     '<tool_response>',
     '</tool_response>',
+    '<question>',
+    '</question>',
     '<think>',
     '</think>',
     '<answer>',