codys12
/

Hunyuen-preview2

Text Generation

Model card Files Files and versions

codys12 commited on Jul 12

Commit

96607d9

·

verified ·

1 Parent(s): 65318aa

Create tokenization_hy.py

Files changed (1) hide show

tokenization_hy.py +1 -1

tokenization_hy.py CHANGED Viewed

@@ -295,4 +295,4 @@ if __name__ == "__main__":
     print(ids)
     text2 = tokenizer.convert_tokens_to_string(tokens)
     print(text2)
-    ids2 = tokenizer.convert_tokens_to_ids(tokens)

     print(ids)
     text2 = tokenizer.convert_tokens_to_string(tokens)
     print(text2)
+    ids2 = tokenizer.convert_tokens_to_ids(tokens)