[CI Fix] Try fixing eagle e2e test OOM by reducing block allocation #20213
Conversation
Signed-off-by: mgoin <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @mgoin, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request addresses Out-Of-Memory (OOM) issues observed in the eagle end-to-end tests by optimizing the GPU memory allocation for specific test configurations. It introduces explicit limits on the number of GPU blocks and maximum model length for the llama2 and llama3 eagle correctness tests, aiming to make these tests more stable and reliable in CI environments.
Highlights
- Test Configuration Optimization: Modified the
test_llama2_eagle_e2e_greedy_correctnessandtest_llama3_eagle_e2e_greedy_correctnessfunctions to explicitly setnum_gpu_blocks_overrideandmax_model_lenparameters. This aims to reduce GPU memory allocation during these specific end-to-end tests, addressing potential Out-Of-Memory (OOM) issues.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request aims to resolve OOM issues in E2E tests by explicitly setting num_gpu_blocks_override and max_model_len. The changes are functionally correct for their stated purpose of reducing memory allocation. My feedback focuses on improving the maintainability of the test configuration.
| "num_gpu_blocks_override": 2 + 256 // 16, | ||
| "max_model_len": (2 + 256 // 16) * 16, |
There was a problem hiding this comment.
The numeric literals 2, 256, and 16 are used directly in the calculations for num_gpu_blocks_override and max_model_len. While the inline comment provides some context, defining these values as named constants (e.g., PROMPT_BLOCK_COUNT, GENERATED_TOKEN_COUNT, KV_CACHE_BLOCK_SIZE) would enhance readability and make the purpose of these numbers explicit. This practice helps prevent errors if the values need to be changed in the future, as it centralizes their definition. Consider defining these constants at a higher scope within the test file.
| # 2 for small prompt, 256//16 for generated. | ||
| "num_gpu_blocks_override": 2 + 256 // 16, | ||
| "max_model_len": (2 + 256 // 16) * 16, |
There was a problem hiding this comment.
The configuration block for common_llm_kwargs, including the num_gpu_blocks_override and max_model_len settings, is duplicated in both test_llama2_eagle_e2e_greedy_correctness and test_llama3_eagle_e2e_greedy_correctness. To improve maintainability and adhere to the DRY (Don't Repeat Yourself) principle, consider defining this common configuration once (e.g., as a shared dictionary or a common fixture) and reusing it across the relevant pytest.mark.parametrize decorators.
mgoin
added
bug
mgoin
changed the title
…llm-project#20213) Signed-off-by: mgoin <[email protected]> Signed-off-by: avigny <[email protected]>
Purpose
FIX #20214
The constantly failing test in CI seems to be an OOM during kv cache allocation, so maybe we can get through by just manually specifying the kv cache blocks for now. I'm not sure if it is a memory profiling issue or just a small GPU issue since the tests have failed for weeks at this point.
Test Plan
See if the spec decode test is green in CI, see #20214
Test Result