New tokenizers support

[dianjixz]

Do you have plans to use the new 的 tokenizers? Transformers are too slow; it takes 10 seconds to start on ax630C. If possible, could you refer to the implementation at: https://github.com/huggingface/tokenizers.git? This one imports very quickly.

[dianjixz]

I tried to write a tokenizer service, which might be useful. You can find it at: https://github.com/m5stack/StackFlow/blob/dev/projects/llm_framework/main_llm_tokenizer/_tokenizer.py

[nnn112358]

Looks like it moved this way.
https://github.com/m5stack/StackFlow/tree/dev/projects/llm_framework/main_llm/scripts