Skip to content

Conversation

vMaroon
Copy link
Member

@vMaroon vMaroon commented Sep 10, 2025

Summary

kvcache.Indexer to return empty-scores if kvblock.TokenProcessor does not generate keys.

Signed-off-by: Maroon Ayoub <[email protected]>
@yankay
Copy link
Collaborator

yankay commented Sep 11, 2025

If the score returns nil, then llm-d-inference-scheduler might also need some adjustments. Add some checks for nil.

https://github.com/llm-d/llm-d-inference-scheduler/blob/main/pkg/plugins/scorer/utils.go#L16

@vMaroon
Copy link
Member Author

vMaroon commented Sep 11, 2025

@yankay agreed that it can be improved there as well - although the current behavior will just be a few useless calls.

@vMaroon
Copy link
Member Author

vMaroon commented Sep 25, 2025

I have tested that this works with the inference-scheduler as-is. Moving in for v0.3.1 due to urgency.

/lgtm
/approve

@github-actions github-actions bot added the lgtm label Sep 25, 2025
@vMaroon vMaroon merged commit 598637a into main Sep 25, 2025
1 check passed
@vMaroon vMaroon deleted the keys branch September 25, 2025 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants