[gpt-oss] add input/output usage in responses api when harmony context is leveraged

[gcalmettes]

Purpose

Dedicated classes for GPT-OSS have been introduced in #22340 in order to handle the specifics of harmony. However currently the token usage statistics are not outputed when serving requests for gpt-oss.

This PR aims to add usage statistics

cc: @WoosukKwon

Test Plan

Test Result

(Optional) Documentation Update

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request adds token usage statistics (num_prompt_tokens and num_output_tokens) to HarmonyContext and its subclasses, which was a missing feature. The changes correctly address the feature request.

My review focuses on improving the maintainability and correctness of the implementation. I've identified a case of code duplication that should be refactored to avoid future bugs. More critically, I've found a potential bug in the streaming context where the token processing and counting logic might be incorrect if more than one token is received in a single output, which can lead to silent errors and incorrect metrics. Addressing these points will make the implementation more robust.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gcalmettes.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[heheda12345]

Please note that when python / browser tool is enabled, there will be multiple rounds of adding new tool call output and generate new tokens. Can you help to handle this scenario correctly? To my understanding, the expect behavior should be sum the prompt length and output length for all rounds. But open to discuss.

And please keep the TODO for num_cached_tokens and num_reasoning_tokens

[gcalmettes]

@heheda12345 good points.
I believe the summing was already done for the output tokens, but was indeed not at all handled for the prompt tokens.

[gcalmettes]

I have updated the changes and it should now be robust to the "inner" multi-turns messages due to built-in tool calls.
I also have verified and indeed the full prompt (previous prompt + concatenation of output from previous messages) is resend each time with each RequestOutput when built-in tools are called and reasoned upon, so the sum has to be done.

[heheda12345]

For streaming case, append_output is called for each output token. Then I think sum up the number of every call to append_output is not correct.

[QierLi]

Wonder if we can count num_cached_tokens and num_reasoning_tokens too?

For cached tokens, we can get the output.num_cached_tokens based on the first output to append.
For reasoning tokens, they can be identified by parser.channel = "analysis".

Also for the input_tokens, I think we should count based on the first output to append, but not the CoT input from the multi-turn output iiuc.

Reference to oai documentation https://platform.openai.com/docs/api-reference/responses/object#responses/object-usage

[heheda12345]

@gcalmettes Can you update the PR for streaming case?

[gcalmettes]

@heheda12345 yes, working on it now, I'll ping you once the changes have been made

[gcalmettes]

@heheda12345 I have pushed a fix for the streaming case. Let me know what you think.

[gcalmettes]

Also for the input_tokens, I think we should count based on the first output to append, but not the CoT input from the multi-turn output iiuc.

@QierLi that is what I was wondering. They are input tokens from the model viewpoint (in the sense that they are ingested as prompt by the model), but not necessarily from the user viewpoint (the user did not generated those subsequent prompts used in the CoT). Would you categorize both the input/output tokens from the CoT as output tokens then ?

[heheda12345]

LGTM!

[heheda12345]

I prefer to count them as input token for every round, because these usage are mainly used for pricing, so it should reflect the compute resource consumption.