[P/D] NIXL Integration #17751

robertgshaw2-redhat · 2025-05-06T23:17:37Z

SUMMARY:

support "dynamo-style" direct KV cache transfer
support fully async send/recv
support runtime NIXL handshake
support xPyD
support homogeneous TP > 1
support P->D request flow

FOLLOW UPS:

support D->P request flow (dynamo-style request flow)
support Heterogeneous TP
support DP attention
robustness to failures
consider more edge cases (prompt logprobs, parallel sampling)
local attention

* [Update] LMcache connector v1 implementation Signed-off-by: ApostaC <[email protected]> * [Add] examples for disaggregated prefill Signed-off-by: ApostaC <[email protected]> * [add] extra information about evns Signed-off-by: ApostaC <[email protected]> * Initial stubs for P/D scheduling changes Signed-off-by: Tyler Michael Smith <[email protected]> * Updates Signed-off-by: Tyler Michael Smith <[email protected]> * Rs branch (#3) * updated Signed-off-by: [email protected] <[email protected]> * Rs branch (#5) Signed-off-by: [email protected] <[email protected]> * Remove Unneeded Arguments (#7) * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * cleanup Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * Improve disagg-example.sh (#8) - fix spelling - CUDA_VISIBLE_DEVICES should be set externally Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * added connector Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * update Signed-off-by: [email protected] <[email protected]> * remove Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * seems to load properly Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * Revert "updated" This reverts commit 97316d9. * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * added Signed-off-by: [email protected] <[email protected]> * diffs for local dev on macos Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * update Signed-off-by: Robert Shaw <[email protected]> * updaed Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Checkpoint. Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * WIP Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated on scheduler side Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Hacking away Signed-off-by: Tyler Michael Smith <[email protected]> * cleanup Signed-off-by: Robert Shaw <[email protected]> * ensure request removed from running list Signed-off-by: Robert Shaw <[email protected]> * Runs E2E. Garbage output. Crashes on 2nd request Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * rename files Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * update Signed-off-by: Robert Shaw <[email protected]> * Second request no longer crashes Signed-off-by: Tyler Michael Smith <[email protected]> * Remove gpu_model_runner hacks Signed-off-by: Tyler Michael Smith <[email protected]> * Clean up Justfile Signed-off-by: Tyler Michael Smith <[email protected]> * [Bugfix] Stale finished requests in EMPTY_MODEL_RUNNER_OUTPUT Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * justfile edits Signed-off-by: Tyler Michael Smith <[email protected]> * Update Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes - lm_eval gsm8k has correctness Signed-off-by: Tyler Michael Smith <[email protected]> * "just delete the assert" Signed-off-by: Tyler Michael Smith <[email protected]> * fixup precommit issues Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes Signed-off-by: Tyler Michael Smith <[email protected]> * updated (#12) Signed-off-by: [email protected] <[email protected]> * Add Accuracy Test (#13) * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * Preemption Bugfixes (#15) * stash fixed double free issue Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * fixed issue Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * updated (#16) Signed-off-by: [email protected] <[email protected]> * Fix Bad Merge | Fix Memory Leak in Upstream (#18) * updated Signed-off-by: [email protected] <[email protected]> * fix merge Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * clean up justfile, examples Signed-off-by: Tyler Michael Smith <[email protected]> * more cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * more cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * more cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * more cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * More cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * more cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * more cleanup, precommit fixes Signed-off-by: Tyler Michael Smith <[email protected]> * More cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * run_accuracy_test.sh UX Signed-off-by: Tyler Michael Smith <[email protected]> * squash warnings Signed-off-by: Tyler Michael Smith <[email protected]> * pre-commit Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * Add get_finished to base kv connector Signed-off-by: mgoin <[email protected]> * revert test.txt Signed-off-by: Tyler Michael Smith <[email protected]> * cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * Cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * review comments Signed-off-by: Tyler Michael Smith <[email protected]> --------- Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]>

Signed-off-by: [email protected] <[email protected]>

* [V1] Support multiple kv connectors Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> * Example script Signed-off-by: mgoin <[email protected]> * . Signed-off-by: mgoin <[email protected]> * Add test Signed-off-by: mgoin <[email protected]> * make mypy happy Signed-off-by: mgoin <[email protected]> * move MultiKVConnectorMetadata to multi_connector.py Signed-off-by: Nick Hill <[email protected]> * minor simplifications Signed-off-by: Nick Hill <[email protected]> * Remove script Signed-off-by: mgoin <[email protected]> * michael inprogress Signed-off-by: Nick Hill <[email protected]> * Make sure we pop requests from connector dict Signed-off-by: mgoin <[email protected]> * req_id -> request_id Signed-off-by: mgoin <[email protected]> --------- Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> Co-authored-by: mgoin <[email protected]>

* Test xPyD Signed-off-by: Tyler Michael Smith <[email protected]> * backwards compatibility Signed-off-by: Tyler Michael Smith <[email protected]> * settable from env Signed-off-by: Tyler Michael Smith <[email protected]> --------- Signed-off-by: Tyler Michael Smith <[email protected]>

To verify load precedence behavior Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Michael Goin <[email protected]>

* [Update] LMcache connector v1 implementation Signed-off-by: ApostaC <[email protected]> * [Add] examples for disaggregated prefill Signed-off-by: ApostaC <[email protected]> * [add] extra information about evns Signed-off-by: ApostaC <[email protected]> * Initial stubs for P/D scheduling changes Signed-off-by: Tyler Michael Smith <[email protected]> * Updates Signed-off-by: Tyler Michael Smith <[email protected]> * Rs branch (#3) * updated Signed-off-by: [email protected] <[email protected]> * Rs branch (#5) Signed-off-by: [email protected] <[email protected]> * Remove Unneeded Arguments (#7) * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * cleanup Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * Improve disagg-example.sh (#8) - fix spelling - CUDA_VISIBLE_DEVICES should be set externally Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * added connector Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * update Signed-off-by: [email protected] <[email protected]> * remove Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * seems to load properly Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * Revert "updated" This reverts commit 97316d9. * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * added Signed-off-by: [email protected] <[email protected]> * diffs for local dev on macos Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * update Signed-off-by: Robert Shaw <[email protected]> * updaed Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Checkpoint. Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * WIP Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated on scheduler side Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Hacking away Signed-off-by: Tyler Michael Smith <[email protected]> * cleanup Signed-off-by: Robert Shaw <[email protected]> * ensure request removed from running list Signed-off-by: Robert Shaw <[email protected]> * Runs E2E. Garbage output. Crashes on 2nd request Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * rename files Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * update Signed-off-by: Robert Shaw <[email protected]> * Second request no longer crashes Signed-off-by: Tyler Michael Smith <[email protected]> * Remove gpu_model_runner hacks Signed-off-by: Tyler Michael Smith <[email protected]> * Clean up Justfile Signed-off-by: Tyler Michael Smith <[email protected]> * [Bugfix] Stale finished requests in EMPTY_MODEL_RUNNER_OUTPUT Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * justfile edits Signed-off-by: Tyler Michael Smith <[email protected]> * Update Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes - lm_eval gsm8k has correctness Signed-off-by: Tyler Michael Smith <[email protected]> * "just delete the assert" Signed-off-by: Tyler Michael Smith <[email protected]> * fixup precommit issues Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes Signed-off-by: Tyler Michael Smith <[email protected]> * updated (#12) Signed-off-by: [email protected] <[email protected]> * Add Accuracy Test (#13) * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * Preemption Bugfixes (#15) * stash fixed double free issue Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * fixed issue Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * updated (#16) Signed-off-by: [email protected] <[email protected]> * Fix Bad Merge | Fix Memory Leak in Upstream (vllm-project#18) * updated Signed-off-by: [email protected] <[email protected]> * fix merge Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * cleanup code Signed-off-by: [email protected] <[email protected]> * cleanup code Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updatted Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * revert Signed-off-by: [email protected] <[email protected]> * more spurious changes Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * Update vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py Co-authored-by: Tyler Michael Smith <[email protected]> * Update vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py Co-authored-by: Tyler Michael Smith <[email protected]> --------- Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Robert Shaw <[email protected]>

* updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * add test Signed-off-by: [email protected] <[email protected]> * add test Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]>

* [Update] LMcache connector v1 implementation Signed-off-by: ApostaC <[email protected]> * [Add] examples for disaggregated prefill Signed-off-by: ApostaC <[email protected]> * [add] extra information about evns Signed-off-by: ApostaC <[email protected]> * Initial stubs for P/D scheduling changes Signed-off-by: Tyler Michael Smith <[email protected]> * Updates Signed-off-by: Tyler Michael Smith <[email protected]> * Rs branch (#3) * updated Signed-off-by: [email protected] <[email protected]> * Rs branch (#5) Signed-off-by: [email protected] <[email protected]> * Remove Unneeded Arguments (#7) * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * cleanup Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * Improve disagg-example.sh (#8) - fix spelling - CUDA_VISIBLE_DEVICES should be set externally Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * added connector Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * update Signed-off-by: [email protected] <[email protected]> * remove Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * seems to load properly Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * Revert "updated" This reverts commit 97316d9. * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * added Signed-off-by: [email protected] <[email protected]> * diffs for local dev on macos Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * update Signed-off-by: Robert Shaw <[email protected]> * updaed Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Checkpoint. Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Cleanup Signed-off-by: Tyler Michael Smith <[email protected]> * WIP Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated on scheduler side Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * Hacking away Signed-off-by: Tyler Michael Smith <[email protected]> * cleanup Signed-off-by: Robert Shaw <[email protected]> * ensure request removed from running list Signed-off-by: Robert Shaw <[email protected]> * Runs E2E. Garbage output. Crashes on 2nd request Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * rename files Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * updated Signed-off-by: Robert Shaw <[email protected]> * update Signed-off-by: Robert Shaw <[email protected]> * Second request no longer crashes Signed-off-by: Tyler Michael Smith <[email protected]> * Remove gpu_model_runner hacks Signed-off-by: Tyler Michael Smith <[email protected]> * Clean up Justfile Signed-off-by: Tyler Michael Smith <[email protected]> * [Bugfix] Stale finished requests in EMPTY_MODEL_RUNNER_OUTPUT Signed-off-by: Tyler Michael Smith <[email protected]> * update Signed-off-by: Tyler Michael Smith <[email protected]> * justfile edits Signed-off-by: Tyler Michael Smith <[email protected]> * Update Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes - lm_eval gsm8k has correctness Signed-off-by: Tyler Michael Smith <[email protected]> * "just delete the assert" Signed-off-by: Tyler Michael Smith <[email protected]> * fixup precommit issues Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes Signed-off-by: Tyler Michael Smith <[email protected]> * updated (#12) Signed-off-by: [email protected] <[email protected]> * Add Accuracy Test (#13) * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * Preemption Bugfixes (#15) * stash fixed double free issue Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * fixed issue Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> * updatrd Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * updated (#16) Signed-off-by: [email protected] <[email protected]> * Fix Bad Merge | Fix Memory Leak in Upstream (#18) * updated Signed-off-by: [email protected] <[email protected]> * fix merge Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * cleanup code Signed-off-by: [email protected] <[email protected]> * cleanup code Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * stash Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updatted Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * revert Signed-off-by: [email protected] <[email protected]> * more spurious changes Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * updated Signed-off-by: [email protected] <[email protected]> * Support MLA in NIXL connector Signed-off-by: Tyler Michael Smith <[email protected]> * WIP adding tests Signed-off-by: Tyler Michael Smith <[email protected]> * wip Signed-off-by: Tyler Michael Smith <[email protected]> * Fixes Signed-off-by: Tyler Michael Smith <[email protected]> --------- Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: Robert Shaw <[email protected]>

Signed-off-by: [email protected] <[email protected]>

Signed-off-by: Tyler Michael Smith <[email protected]>

…ge_main [2025-5-5] Upstream sync

Signed-off-by: [email protected] <[email protected]>

github-actions · 2025-05-06T23:17:46Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: [email protected] <[email protected]>

kouroshHakha · 2025-05-12T15:32:39Z

@robertgshaw2-redhat question: Does P instance block for handshake to happen? The proxy suggests that prefill does not need to know about the decode instance when the request is sent to it. Can you at high level explain the data and control flow of this architecture?

heheda12345 · 2025-05-12T16:19:54Z

vllm/v1/core/kv_cache_manager.py

+            for block in self.single_type_manager.req_to_blocks[request_id]
+        ]
+
+    def get_num_blocks(self, request_id: str):


Seems that this function is not used?

you're right, I will remove this.

robertgshaw2-redhat · 2025-05-12T17:46:41Z

@robertgshaw2-redhat question: Does P instance block for handshake to happen? The proxy suggests that prefill does not need to know about the decode instance when the request is sent to it. Can you at high level explain the data and control flow of this architecture?

Here's the doc:

https://docs.google.com/document/d/1lDQt6hUXBoMnMabsuAMZAWdqit7TzuflsFGHLt2gZus/edit?tab=t.0

kouroshHakha · 2025-05-12T17:48:27Z

@robertgshaw2-redhat thanks, requested access.

The KVConnector API have been updated after merging the integration of NIXL in this PR: vllm-project/vllm#17751 Update VUA's method signatures to match the new API. Signed-off-by: diabloneo <[email protected]>

Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Brent Salisbury <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Brent Salisbury <[email protected]>

Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Brent Salisbury <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Brent Salisbury <[email protected]> Signed-off-by: Yuqi Zhang <[email protected]>

LCAIZJ · 2025-05-24T14:58:35Z

@robertgshaw2-redhat Can you open access to this document https://docs.google.com/document/d/1lDQt6hUXBoMnMabsuAMZAWdqit7TzuflsFGHLt2gZus/edit?tab=t.0

zhaohaidao · 2025-05-25T16:01:28Z

@robertgshaw2-redhat question: Does P instance block for handshake to happen? The proxy suggests that prefill does not need to know about the decode instance when the request is sent to it. Can you at high level explain the data and control flow of this architecture?

Here's the doc:

https://docs.google.com/document/d/1lDQt6hUXBoMnMabsuAMZAWdqit7TzuflsFGHLt2gZus/edit?tab=t.0

@robertgshaw2-redhat Hi，Can you grant me access? requested access.

new-TonyWang · 2025-06-10T08:57:36Z

hello, and execuse me. When I was learning NIXL intergration, I have a question about this pr. How does WAITING_FOR_REMOTE_KVS work in PD Disaggregation and why is this different from other solution (like simple connector). Looking forward to your reply. Thank you very much.

Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Brent Salisbury <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Brent Salisbury <[email protected]> Signed-off-by: minpeter <[email protected]>

AsicDyc · 2025-07-31T08:29:23Z

@robertgshaw2-redhat question: Does P instance block for handshake to happen? The proxy suggests that prefill does not need to know about the decode instance when the request is sent to it. Can you at high level explain the data and control flow of this architecture?

Here's the doc:

https://docs.google.com/document/d/1lDQt6hUXBoMnMabsuAMZAWdqit7TzuflsFGHLt2gZus/edit?tab=t.0

Hey, can you give me a request access?

david6666666 · 2025-07-31T08:59:30Z

@robertgshaw2-redhat question: Does P instance block for handshake to happen? The proxy suggests that prefill does not need to know about the decode instance when the request is sent to it. Can you at high level explain the data and control flow of this architecture?

Here's the doc:

https://docs.google.com/document/d/1lDQt6hUXBoMnMabsuAMZAWdqit7TzuflsFGHLt2gZus/edit?tab=t.0

Hey, can you give me a request access?

tlrmchlsmth and others added 13 commits May 3, 2025 11:13

fix failing tests (vllm-project#64)

c90a2c8

Signed-off-by: [email protected] <[email protected]>

[Test] Improve MultiConnector test (vllm-project#69)

8061a5c

To verify load precedence behavior Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Michael Goin <[email protected]>

[Bugfix] Fix env name for VLLM_NIXL_SIDE_CHANNEL_HOST (vllm-project#73)

8ce4c07

Signed-off-by: [email protected] <[email protected]>

Merge branch 'main' into disagg_pd_dev

bf0be1b

Signed-off-by: Tyler Michael Smith <[email protected]>

Merge pull request vllm-project#76 from neuralmagic/disagg_pd_dev_mer…

3783696

…ge_main [2025-5-5] Upstream sync

updated

42b869e

Signed-off-by: [email protected] <[email protected]>

mergify bot added ci/build frontend v1 labels May 6, 2025

robertgshaw2-redhat added 13 commits May 6, 2025 23:19

fixup testing

bb2abeb

Signed-off-by: [email protected] <[email protected]>

remove multi-connector

b2de5e9

Signed-off-by: [email protected] <[email protected]>

remove multi-connector

391a94a

Signed-off-by: [email protected] <[email protected]>

remove multi-connector

42d6d26

Signed-off-by: [email protected] <[email protected]>

cleanup paths

527fbf1

Signed-off-by: [email protected] <[email protected]>

cleanup paths

40fd5b0

Signed-off-by: [email protected] <[email protected]>

cleanup paths

79561f4

Signed-off-by: [email protected] <[email protected]>

cleanup paths

ccd356a

Signed-off-by: [email protected] <[email protected]>

cleanup paths

43adf8e

Signed-off-by: [email protected] <[email protected]>

cleanup paths

e98b512

Signed-off-by: [email protected] <[email protected]>

cleanup paths

84c1379

Signed-off-by: [email protected] <[email protected]>

cleanup

e3e1738

Signed-off-by: [email protected] <[email protected]>

cleanup

cd2484d

Signed-off-by: [email protected] <[email protected]>

robertgshaw2-redhat added 5 commits May 12, 2025 01:09

mypy

f665786

Signed-off-by: [email protected] <[email protected]>

added edge case tests

626bb11

Signed-off-by: [email protected] <[email protected]>

added edge case tests

4f0a5e9

Signed-off-by: [email protected] <[email protected]>

added edge case tests

31bf313

Signed-off-by: [email protected] <[email protected]>

added edge case tests

56937cd

Signed-off-by: [email protected] <[email protected]>

NickLucche mentioned this pull request May 12, 2025

[P/D] Heterogenous TP #17990

Closed

updated

42b1ad0

Signed-off-by: [email protected] <[email protected]>

heheda12345 reviewed May 12, 2025

View reviewed changes

simon-mo merged commit d191102 into vllm-project:main May 12, 2025
72 of 87 checks passed

diabloneo mentioned this pull request May 13, 2025

Update API signatures according to new KVConnector design vast-data/VUA#2

Merged

NickLucche mentioned this pull request May 15, 2025

[PD] Heterogenous TP + #7 robertgshaw2-redhat/vllm#14

Closed

juncgu mentioned this pull request May 17, 2025

[P/D] Support CPU Transfer in NixlConnector #18293

Merged

lk-chen mentioned this pull request May 20, 2025

[serve.llm] Prefill-Decode disaggregation initial implementation ray-project/ray#53092

Merged

kouroshHakha mentioned this pull request May 23, 2025

[RFC][llm] Prefill/Decode disaggregation with Ray Serve ray-project/ray#53257

Open

rainj-me mentioned this pull request Jun 6, 2025

[RFC][Feature] Support Remote Prefill in PD Disaggregation sgl-project/sglang#6925

Open

2 tasks

sdavidbd mentioned this pull request Jun 8, 2025

[RFC]: Graceful Error Handling for KV Connector Load Failures #19329

Open

1 task

jianzs mentioned this pull request Jun 20, 2025

[Feature]: kv_transfer_params not returned for multiple subrequests #19902

Open

1 task

tanujtiwari1998 mentioned this pull request Jul 8, 2025

cached tokens completions character-tech/vllm#22

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[P/D] NIXL Integration #17751

[P/D] NIXL Integration #17751

robertgshaw2-redhat commented May 6, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented May 6, 2025

Uh oh!

kouroshHakha commented May 12, 2025

Uh oh!

heheda12345 May 12, 2025

Uh oh!

robertgshaw2-redhat May 12, 2025

Uh oh!

Uh oh!

robertgshaw2-redhat commented May 12, 2025

Uh oh!

kouroshHakha commented May 12, 2025

Uh oh!

LCAIZJ commented May 24, 2025

Uh oh!

zhaohaidao commented May 25, 2025

Uh oh!

new-TonyWang commented Jun 10, 2025

Uh oh!

AsicDyc commented Jul 31, 2025

Uh oh!

david6666666 commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

[P/D] NIXL Integration #17751

[P/D] NIXL Integration #17751

Conversation

robertgshaw2-redhat commented May 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 6, 2025

Uh oh!

kouroshHakha commented May 12, 2025

Uh oh!

heheda12345 May 12, 2025

Choose a reason for hiding this comment

Uh oh!

robertgshaw2-redhat May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robertgshaw2-redhat commented May 12, 2025

Uh oh!

kouroshHakha commented May 12, 2025

Uh oh!

LCAIZJ commented May 24, 2025

Uh oh!

zhaohaidao commented May 25, 2025

Uh oh!

new-TonyWang commented Jun 10, 2025

Uh oh!

AsicDyc commented Jul 31, 2025

Uh oh!

david6666666 commented Jul 31, 2025

Uh oh!

Uh oh!

robertgshaw2-redhat commented May 6, 2025 •

edited by github-actions bot

Loading