[Feature] Tokenizer endpoint in server mode

[junjzhang]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Using Server mode to generate Rollout in Agentic RL training is a very necessary and natural approach. However, the design of Agent Scaffold typically only considers compatibility with OpenAI compatible API interface, making it difficult to collect token IDs at the Agent Scaffold level—information that is essential for training. Additionally, current design couples tokenization with the inference model, which indicates it's a logically sound idea to let inference engine handle tokenization.
Thus, a tokenize and endpoint is needed.

Related resources

Maybe refer to vllm's tokenize endpoint. https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#tokenizer-api

[adarshxs]

On it!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.