Skip to content

🚀 Scaling GRPO to 70B+ Models and Multi-Node Training with vLLM Server & NCCL Communication #3094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
qgallouedec merged 81 commits into from
Mar 21, 2025
Merged

🚀 Scaling GRPO to 70B+ Models and Multi-Node Training with vLLM Server & NCCL Communication #3094

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
765891c
🚀allow GRPO to connect to VLLM in remote/local node with NCCL communi…
binary-husky Mar 16, 2025
42f2131
Update trl/extras/remote_vllm_helper.py
binary-husky Mar 17, 2025
715d486
use argparse for options
kashif Mar 17, 2025
60a6753
add imports for remote vllm helper
kashif Mar 17, 2025
f784a8c
formatting
kashif Mar 17, 2025
5628b60
fix arguments
kashif Mar 17, 2025
8bfc313
use cli options
kashif Mar 17, 2025
d63c94a
vllm serve
qgallouedec Mar 18, 2025
c2e970f
clean server
qgallouedec Mar 18, 2025
e50d288
better naming
qgallouedec Mar 18, 2025
c723685
client
qgallouedec Mar 18, 2025
5d19cf1
style
qgallouedec Mar 18, 2025
b5ff472
new params in generate
qgallouedec Mar 18, 2025
e5fe142
this method is the new default
qgallouedec Mar 18, 2025
73853fc
update config
qgallouedec Mar 18, 2025
1fbdf69
Merge branch 'main' into main
qgallouedec Mar 18, 2025
94625f9
do not use asserts
kashif Mar 18, 2025
9335e68
update config
qgallouedec Mar 18, 2025
06aca0a
separate host and post
qgallouedec Mar 18, 2025
a7af2e2
Merge branch 'main' of https://github.com/binary-husky/trl into pr/bi…
qgallouedec Mar 18, 2025
a92b296
proper deprectation
qgallouedec Mar 18, 2025
714a833
deprecated arg in the vllm server
qgallouedec Mar 18, 2025
71024d6
simplify moving
qgallouedec Mar 18, 2025
bbf99f1
document host and port
qgallouedec Mar 18, 2025
a7e9dea
style
qgallouedec Mar 18, 2025
2b7fb1a
update trainer
qgallouedec Mar 18, 2025
5fee194
new generate args
qgallouedec Mar 18, 2025
508bd90
update doc
qgallouedec Mar 19, 2025
75bd4e3
Fix for zero3
qgallouedec Mar 19, 2025
5a8138c
Better naming
qgallouedec Mar 19, 2025
5f19c70
Remove remote_vllm_helper
qgallouedec Mar 19, 2025
4ae6cb4
remove grpo_with_remote_vllm
qgallouedec Mar 19, 2025
9ca4dde
remove cloudpickle from deps
qgallouedec Mar 19, 2025
5d1398e
Some consistency
qgallouedec Mar 19, 2025
e85c7bb
Merge branch 'main' into main
qgallouedec Mar 19, 2025
44ae792
Update docs/source/grpo_trainer.md
kashif Mar 19, 2025
060c4a6
Update setup.py
kashif Mar 19, 2025
d581a5f
add revision argument to vllm server
kashif Mar 19, 2025
128b503
Update docs/source/grpo_trainer.md
kashif Mar 19, 2025
724e013
Update docs/source/grpo_trainer.md
kashif Mar 19, 2025
daf2cde
Reset the prefix cache after updating weights
kashif Mar 19, 2025
75bfbc4
Merge remote-tracking branch 'refs/remotes/binary-husky/main'
kashif Mar 19, 2025
bb1fb55
Update vllm_client.py
qgallouedec Mar 19, 2025
415f3ca
Update vllm_client.py
qgallouedec Mar 19, 2025
1053197
Update vllm_serve.py
qgallouedec Mar 19, 2025
e6a4901
Add health check endpoint to vLLM server
qgallouedec Mar 19, 2025
e763064
connection timeout
qgallouedec Mar 19, 2025
4554af9
style
qgallouedec Mar 19, 2025
92a154f
fix doc langauge hint
qgallouedec Mar 19, 2025
666a6e4
Merge branch 'main' into main
kashif Mar 20, 2025
6537a7e
move reset_prefix_cache to its own endpoint
kashif Mar 20, 2025
821d37e
Merge branch 'main' of https://github.com/binary-husky/trl into pr/bi…
qgallouedec Mar 20, 2025
c38e79f
async
qgallouedec Mar 20, 2025
92cf3e0
merge peft adaptor to send to vllm
kashif Mar 20, 2025
0ffec9f
Looks simple. Wasn't.
qgallouedec Mar 20, 2025
d9d28db
Peft compatibility
qgallouedec Mar 21, 2025
d452a2f
Update docs/source/speeding_up_training.md
kashif Mar 21, 2025
dd873cf
Update docs/source/speeding_up_training.md
kashif Mar 21, 2025
7e11184
Update trl/extras/vllm_client.py
kashif Mar 21, 2025
6c4bf00
GatheredParameters can be disabled
kashif Mar 21, 2025
c431b0f
gather and ungather peft weights within the same deepseed context
kashif Mar 21, 2025
67c4e68
use is_vllm_available
kashif Mar 21, 2025
15fcaaf
minor consistency fixes
qgallouedec Mar 21, 2025
09ec2a1
fix error when deepspeed is not installed
kashif Mar 21, 2025
bc2f902
fix deepspeed import when not peft
kashif Mar 21, 2025
db8d5fd
Merge branch 'main' of https://github.com/binary-husky/trl into pr/bi…
qgallouedec Mar 21, 2025
89812ee
simpler
kashif Mar 21, 2025
bb66c91
multinode doc
qgallouedec Mar 21, 2025
b23c23f
minor code and comments changes
qgallouedec Mar 21, 2025
5111c8f
Merge branch 'main' of https://github.com/binary-husky/trl into pr/bi…
qgallouedec Mar 21, 2025
657cb21
style
qgallouedec Mar 21, 2025
8670c35
optional deps
qgallouedec Mar 21, 2025
7955a39
vllm_server_timeout as arg
qgallouedec Mar 21, 2025
5a37647
small refinement in doc
qgallouedec Mar 21, 2025
10d26ef
update deps
qgallouedec Mar 21, 2025
d759c9c
Fix VLLMClient argument in grpo_trainer; Add zero3+peft vllm transfer…
binary-husky Mar 21, 2025
4fc8790
Revert "Fix VLLMClient argument in grpo_trainer; Add zero3+peft vllm …
qgallouedec Mar 21, 2025
fb28f62
log num_tokens
qgallouedec Mar 21, 2025
3a211e6
disable vllm test (in the future we'll add a mock for vllm server for…
qgallouedec Mar 21, 2025
7a81655
style
qgallouedec Mar 21, 2025
716a822
fix ds3_gather_for_generation
qgallouedec Mar 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading