[Core] Add update_config RPC method

[22quinn]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Part of #19886

Test Plan

# Create an LLM without loading real weights
llm = LLM(
    model="facebook/opt-125m",
    load_format="dummy",
    enforce_eager=True,
)

llm.collective_rpc("update_config", args=({"load_config": {"load_format": "auto"}},))
llm.collective_rpc("load_model")

# Check real weights are loaded
outputs = llm.generate(prompts, sampling_params)
print("\nGenerated Outputs:\n" + "-" * 60)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt:    {prompt!r}")
    print(f"Output:    {generated_text!r}")
    print("-" * 60)

Test Result

Generated Outputs:
------------------------------------------------------------
Prompt:    'Hello, my name is'
Output:    ' Joel, my dad is my friend and we are in a relationship. I am'
------------------------------------------------------------
Prompt:    'The president of the United States is'
Output:    ' reportedly being investigated by the FBI over the allegations that he broke campaign finance laws in'
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    ' all set to open a new gateway to the most notorious terror camps in Europe,'
------------------------------------------------------------
Prompt:    'The future of AI is'
Output:    ' in their hands. We cannot prevent the future.\nAgreed.  '
------------------------------------------------------------

(Optional) Documentation Update

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Summary of Changes

Hello @22quinn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new RPC method, update_config, designed to provide greater flexibility in managing model configurations within the vLLM framework. This enhancement allows for programmatic adjustments to various configuration objects on both GPU and TPU model runners, enabling dynamic changes such as altering model loading behavior after initialization but prior to the actual model weights being loaded.

Highlights

• New RPC Method: I've added a new RPC method, update_config, to the core vLLM system. This method allows for dynamic modification of configuration objects (e.g., load_config) on ModelRunner instances.
• Configuration Update Mechanism: The update_config method is implemented to iterate through provided overrides, retrieve the existing configuration object, and then use dataclasses.replace to create a new configuration with the specified changes. This ensures that configuration objects are updated immutably and correctly.
• Worker Integration: The GPUWorker and TPUWorker classes now expose the update_config method, which simply delegates the call to their respective model_runner instances. This makes the new configuration update functionality accessible at the worker level, enabling scenarios like updating load_format before model loading.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request introduces an update_config RPC method to both GPU and TPU model runners, allowing for dynamic updates to configuration parameters at runtime. The code includes basic error handling for unknown configurations but lacks type validation for the override values, which could lead to runtime issues. Consider adding type validation to ensure the override values match the expected types.

[aarnphm]

It seems like we might want to merge recursively here. might be good to have a utils for it to implement deepmerge 🤔

[aarnphm]

some comments

[houseroad]

Otherwise, looks good to me.

[ProExpertProg]

Do we have tests that this works correctly with quantization and torch compilation?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @22quinn.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[houseroad]

why the update_config logic is different from gpu one?

[22quinn]

updated, missed it after a few updates. @Chenyaaang @yaochengji perhaps worth refactoring to inherit from a base class as TPU code shares a lot common logic

[yaochengji]

I think maybe different hardware has its own specific logic of checking the configuration. E.g. https://github.com/vllm-project/vllm/blob/main/vllm/platforms/tpu.py#L99

We can add a TODO here

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @22quinn.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[22quinn]

Do we have tests that this works correctly with quantization and torch compilation?

@ProExpertProg I've tested torch.compile works fine via: https://gist.github.com/22quinn/c2ace183a56d0801e160cb133b436bd1
However, quantization is not supported for now. There are some users asking for it. I'll dig a bit and see how to support it in a separate PR.
For now, I want to finish up the basic in-place update functionality. Once basic features are complete and APIs are stable, I'll add an E2E test and example.