Skip to content

[GPTQ] Add inversion fallback #1283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 1, 2025
Merged

Conversation

kylesayrs
Copy link
Collaborator

Purpose

  • Given the increasing size of large language models (such as DeepSeek-R1 which contains 45034 linear layers), the likelihood that any of the hessian inversions will spontaneously fail is significant
  • These changes cause the GPTQ algorithm to fall back to RTN for any layers which fail hessian inversion

Changes

  • Implement fallback by setting hessian value to identity matrix if inversion fails

Signed-off-by: Kyle Sayers <[email protected]>
Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

Copy link
Collaborator

@brian-dellabetta brian-dellabetta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool!

@kylesayrs kylesayrs added the ready When a PR is ready for review label Mar 27, 2025
rahul-tuli
rahul-tuli previously approved these changes Apr 1, 2025
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs kylesayrs dismissed stale reviews from rahul-tuli and brian-dellabetta via 56a5f14 April 1, 2025 16:44
@dsikka dsikka enabled auto-merge (squash) April 1, 2025 16:53
@dsikka dsikka merged commit 0873f28 into main Apr 1, 2025
8 checks passed
@dsikka dsikka deleted the kylesayrs/gptq-inversion-fallback branch April 1, 2025 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready When a PR is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants