[Feature] Add /tokenize and /detokenize OpenAI-compatible endpoints

[adarshxs]

Motivation

requested in: #5653

Adds /tokenize and /detokenize endpoints to the OpenAI-compatible API server.

allows users to directly access the server's tokenizer for encoding text into token IDs and decoding token IDs back into text.

• Implements TokenizeRequest/Response and DetokenizeRequest/Response Pydantic models.
• Adds handler logic in adapter.py utilizing the TokenizerManager.
• Routes are added in http_server.py.
• Supports single/batch inputs and options like add_special_tokens/skip_special_tokens.
• Includes /v1/ path aliases.

works like this:

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.

[merrymercy]

Can you add a test case?

[adarshxs]

Done @merrymercy

[halaction]

Possible updates on this PR? Test failures don't seem to be related to this issue, and the added test cases are successful - i think it's due for a review. This feature would be really helpful.

[adarshxs]

@zhyncs @merrymercy @zhaochenyang20 please review

[FeliceSchena]

Is this PR still under review? It would be helpful to have it merged.

[thigger]

Unfortunately there have been a few breaking changes to the OAI code so I've had to downgrade to 0.4.6-post5 to use this (and a tokenize endpoint is really useful!)

[adarshxs]

Yeah the entire openai api interface has been refactored. I'll try implementing these endpoints for the new refractor ASAP and soon close this PR

[thigger]

Thanks - your patch is the main reason I've been able to use sglang for my usecase!

[Zyann7]

Thanks for your work, this will be really helpful! Is there an estimated timeline for merge or release?

[adarshxs]

cc @thigger @Zyann7 @FeliceSchena @halaction apologies for the delay. opened a new PR with this feature compatible with the new openai interface. Hoping to have it merged soon hence i'll be closing this

[thigger]

Thanks! I've been holding off upgrading sglang for this reason and will pull your new patch