[Quantization] Bump to use latest bitsandbytes #20424

jeejeelee · 2025-07-03T07:57:47Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Signed-off-by: Jee Jee Li <[email protected]>

gemini-code-assist

Summary of Changes

Hello @jeejeelee, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on upgrading the bitsandbytes dependency to version 0.46.1. This update ensures that the project leverages the latest features and fixes from the bitsandbytes library, which is crucial for quantization functionalities. The changes span across various configuration files, documentation, and internal version checks to align the entire codebase with this new dependency version.

Highlights

Dependency Update: Bumped the bitsandbytes library version across the project's dependencies and internal version checks to 0.46.1.
Configuration and Documentation Alignment: Updated Dockerfile, documentation, and various requirements files to reflect the new bitsandbytes version. Also, internal version checks and related comments in the vllm source code were updated to ensure consistency.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This PR updates the bitsandbytes dependency to version 0.46.1 across the project. The changes are consistent and look good. I've added suggestions to refactor the duplicated version check logic by using constants for the version string and install requirement string, which will improve maintainability for future updates.

gemini-code-assist · 2025-07-03T08:00:06Z

vllm/model_executor/layers/quantization/bitsandbytes.py

+            if bitsandbytes.__version__ < "0.46.1":
                raise ImportError("bitsandbytes version is wrong. Please "
-                                  "install bitsandbytes>=0.45.3.")
+                                  "install bitsandbytes>=0.46.1.")


Consider defining 0.46.1 as a constant to avoid hardcoding the version string¹. This can improve maintainability.

Style Guide References

Suggested change

if bitsandbytes.__version__ < "0.46.1":

raise ImportError("bitsandbytes version is wrong. Please "

"install bitsandbytes>=0.45.3.")

"install bitsandbytes>=0.46.1.")

_MIN_VERSION = "0.46.1"

if bitsandbytes.__version__ < _MIN_VERSION:

raise ImportError("bitsandbytes version is wrong. Please "

f"install bitsandbytes>={_MIN_VERSION}.")

Footnotes

Using constants for frequently used values improves readability and maintainability. (link) ↩

gemini-code-assist · 2025-07-03T08:00:06Z

vllm/model_executor/layers/quantization/bitsandbytes.py

+            raise ImportError("Please install bitsandbytes>=0.46.1 via "
+                              "`pip install bitsandbytes>=0.46.1` to use "
                              "bitsandbytes quantizer.") from err


Consider defining bitsandbytes>=0.46.1 as a constant to avoid hardcoding the install requirement string¹.

_MIN_VERSION = "0.46.1" _INSTALL_REQ = f"bitsandbytes>={_MIN_VERSION}" except ImportError as err: raise ImportError( f"Please install {_INSTALL_REQ} via " f"`pip install {_INSTALL_REQ}` to use " "bitsandbytes quantizer.") from err

Style Guide References

Footnotes

Using constants for frequently used values improves readability and maintainability. (link) ↩

gemini-code-assist · 2025-07-03T08:00:07Z

vllm/model_executor/model_loader/bitsandbytes_loader.py

+            if bitsandbytes.__version__ < "0.46.1":
                raise ImportError("bitsandbytes version is wrong. Please "
-                                  "install bitsandbytes>=0.45.3.")
+                                  "install bitsandbytes>=0.46.1.")


Consider defining 0.46.1 as a constant to avoid hardcoding the version string¹. This can improve maintainability.

Style Guide References

Suggested change

if bitsandbytes.__version__ < "0.46.1":

raise ImportError("bitsandbytes version is wrong. Please "

"install bitsandbytes>=0.45.3.")

"install bitsandbytes>=0.46.1.")

_MIN_VERSION = "0.46.1"

if bitsandbytes.__version__ < _MIN_VERSION:

raise ImportError("bitsandbytes version is wrong. Please "

f"install bitsandbytes>={_MIN_VERSION}.")

Footnotes

Using constants for frequently used values improves readability and maintainability. (link) ↩

gemini-code-assist · 2025-07-03T08:00:07Z

vllm/model_executor/model_loader/bitsandbytes_loader.py

+            raise ImportError("Please install bitsandbytes>=0.46.1 via "
+                              "`pip install bitsandbytes>=0.46.1` to use "
                              "bitsandbytes quantizer.") from err


Consider defining bitsandbytes>=0.46.1 as a constant to avoid hardcoding the install requirement string¹.

_MIN_VERSION = "0.46.1" _INSTALL_REQ = f"bitsandbytes>={_MIN_VERSION}" except ImportError as err: raise ImportError("bitsandbytes version is wrong. Please " f"install {_INSTALL_REQ}.")

Style Guide References

Footnotes

Using constants for frequently used values improves readability and maintainability. (link) ↩

github-actions · 2025-07-03T08:07:06Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Jee Jee Li <[email protected]>

Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: avigny <[email protected]>

Signed-off-by: Jee Jee Li <[email protected]>

Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>

jeejeelee added 2 commits July 3, 2025 07:55

Done

65333ca

Signed-off-by: Jee Jee Li <[email protected]>

fix

6744202

Signed-off-by: Jee Jee Li <[email protected]>

jeejeelee requested review from hmellor, simon-mo, WoosukKwon, youkaichao, robertgshaw2-redhat, mgoin, tlrmchlsmth and houseroad as code owners July 3, 2025 07:57

jeejeelee added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 3, 2025

gemini-code-assist bot reviewed Jul 3, 2025

View reviewed changes

jeejeelee requested a review from Isotr0py July 3, 2025 07:58

mergify bot added documentation Improvements or additions to documentation ci/build labels Jul 3, 2025

gemini-code-assist bot reviewed Jul 3, 2025

View reviewed changes

Isotr0py approved these changes Jul 3, 2025

View reviewed changes

DarkLight1337 merged commit 1819fbd into vllm-project:main Jul 3, 2025
114 checks passed

jeejeelee deleted the upgrade-bnb branch July 3, 2025 15:33

sfeng33 pushed a commit to sfeng33/vllm that referenced this pull request Jul 6, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

c22814f

Signed-off-by: Jee Jee Li <[email protected]>

huydhn pushed a commit to huydhn/vllm that referenced this pull request Jul 8, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

1407e7d

Signed-off-by: Jee Jee Li <[email protected]>

Chen-zexi pushed a commit to Chen-zexi/vllm that referenced this pull request Jul 13, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

8400732

Signed-off-by: Jee Jee Li <[email protected]>

LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

a7756b4

Signed-off-by: Jee Jee Li <[email protected]>

avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

0a5f0f4

Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: avigny <[email protected]>

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

2de77fc

Signed-off-by: Jee Jee Li <[email protected]>

npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

6e3ba75

Signed-off-by: Jee Jee Li <[email protected]>

jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025

[Quantization] Bump to use latest bitsandbytes (vllm-project#20424)

0839ab5

Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Quantization] Bump to use latest bitsandbytes #20424

[Quantization] Bump to use latest bitsandbytes #20424

Uh oh!

jeejeelee commented Jul 3, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 3, 2025

Uh oh!

gemini-code-assist bot Jul 3, 2025

Uh oh!

gemini-code-assist bot Jul 3, 2025

Uh oh!

gemini-code-assist bot Jul 3, 2025

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Quantization] Bump to use latest bitsandbytes #20424

[Quantization] Bump to use latest bitsandbytes #20424

Uh oh!

Conversation

jeejeelee commented Jul 3, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 3, 2025

Choose a reason for hiding this comment

Style Guide References

Footnotes

Uh oh!

gemini-code-assist bot Jul 3, 2025

Choose a reason for hiding this comment

Style Guide References

Footnotes

Uh oh!

gemini-code-assist bot Jul 3, 2025

Choose a reason for hiding this comment

Style Guide References

Footnotes

Uh oh!

gemini-code-assist bot Jul 3, 2025

Choose a reason for hiding this comment

Style Guide References

Footnotes

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

jeejeelee commented Jul 3, 2025 •

edited by github-actions bot

Loading