tokenizer suggestions

[davidkoski]

I have a couple of suggestions for the tokenizer API -- things that I have needed to work around here: https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/LLM/Tokenizer.swift

• add eosToken / eosTokenId to the Tokenizer protocol

  • this is needed to know when to stop producing tokens
  • Tokenizer already has unknownToken
  • I don't know if any of the other special tokens should be exposed, e.g. bosToken

• have a way to add to TokenizerModel/knownTokenizers or otherwise handle unknown tokenizers

  • right now it would probably be sufficient to map to "PreTrainedTokenizer": BPETokenizer.self
  • but in the future this might need to be more flexible
  • TokenizerModel is internal as are the various classes like BPETokenizer
  • in my workaround I mapped string -> string, e.g. "Qwen2Tokenizer": "PreTrainedTokenizer", which is perhaps the right level -- not exposing too much of the implementation
  • anyway, some kind of API to allow registration of overrides like this or perhaps just "PreTrainedTokenizer" as a fallback for now

If these fit in with the vision for the tokenizer API, please consider them!

Thanks

[pcuenca]

• Expose eos: +1, I saw your use in mlx-swift and it makes sense. I'll open a PR soon.
• Also thought about the fallback tokenizer method but thought it was better to explicitly fail on unsupported tokenizers. A mechanism for the user to register overrides does look interesting!

[pcuenca]

Opened #70 and ml-explore/mlx-swift-examples#28 to address the first topic.