Skip to content

Fix OpenAI server completion_tokens referenced before assignment #1996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 10, 2023

Conversation

js8544
Copy link
Contributor

@js8544 js8544 commented Dec 9, 2023

When a models return an empty result, the OpenAI server will report an error: local variable completion_tokens referenced before assignment.

@simon-mo
Copy link
Collaborator

simon-mo commented Dec 9, 2023

Good catch. Can you set the completion tokens to 0 in the else clause instead? I believe it is only used in the UsageInfo.

Like UsageInfo(completion_tokens=0).

Thank you!

@js8544
Copy link
Contributor Author

js8544 commented Dec 9, 2023

I don't think using UsageInfo(completion_tokens=0) is correct though. For a non-empty output completion_tokens should be the value set in the if clause.
I do however find out we can use previous_num_tokens[i] instead, to make the code simpler. Please checkout the newest patch.

@simon-mo simon-mo merged commit 1aa1361 into vllm-project:main Dec 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants